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Abstract 


The Bill and Melinda Gates Foundation invested in the Literacy Design Collaborative (LDC) 
as one strategy to support teachers’ and students’ transition to the Common Core State 
Standards (CCSS) in English language arts. This report provides an early look at the 
implementation of LDC in eighth-grade history/social studies and science classes in two 
states, and the effectiveness of the intervention in these settings. The study found that across 
states and subjects, teachers understood LDC and implemented it with fidelity. Teachers also 
generally reported positive attitudes about the effectiveness of LDC and its usefulness in 
introducing literacy instruction into content area classrooms. Quasi-experimental analyses 
using Coarsened Exact Matching (CEM) techniques and hierarchical linear modeling (HLM) 
found a small statistically significant positive effect on reading scores in the one state where 
suitable data were available, but no effects on writing scores. However, students generally 
performed at low levels on assessments designed to align with the intervention, suggesting 
the challenge of meeting CCSS expectations. Exploratory analyses suggest that LDC may 
have been most effective for higher achieving students. However understandable, the findings 
thus suggest that, in the absence of additional scaffolding and supports for low-achieving 
students, LDC may be gap enhancing. 


Chapter 1: Introduction 


The Common Core State Standards (CCSS) in English language arts (ELA) bring rigorous, 
new demands for student accomplishment to ensure that students will have the literacy 
knowledge and skills they need to be prepared for success in college and careers. For most states, 
these new English language arts standards dramatically increase expectations for students’ ability 
to read literary and informational texts closely, analyze evidence, communicate orally and in 


writing for a variety of audiences and purposes, and conduct research. 


The new standards bring with them requirements for pedagogical shifts and challenges for 
teachers who are expected to support their students’ accomplishment of these more rigorous 
goals. The challenge extends not only to elementary school teachers and secondary English 
teachers, those who historically have been charged with students’ literacy development, but to 
secondary content-area teachers as well. That is, the CCSS specifically encompasses literacy 


standards for middle and high school coursework in history/social studies, science, and technical 


subjects. Secondary school content-area teachers now are expected to integrate literacy 
development with their content goals and thus to engage students in curriculum and instruction 


that simultaneously support student learning in both domains. 


The Bill and Melinda Gates Foundation invested in the Literacy Design Collaborative 
(LDC) as one strategy to support teachers’ and students’ transition to these new expectations. 
Although LDC is at a relatively early stage of implementation, the Foundation was interested in 
getting an early read on program effectiveness and contracted with the National Center for 
Research on Evaluation, Standards, and Student Testing (CRESST) to conduct two quasi- 
experimental studies of LDC’s implementation and learning impact. The first study, which is 
reported here, examined LDC as it was implemented in eighth-grade history/social studies and 
science classes during the 2012-2013 school year in selected districts from Kentucky and 
Pennsylvania. The second study, reported in a separate companion report (Herman et al., 2015), 
examines LDC effects in a districtwide implementation in sixth-grade Advanced Reading 


courses in a large countywide district in Florida. 


This chapter presents background on the study, including a brief description of the LDC 
intervention and the evaluation questions that guided the eighth-grade study. In the following 
chapters, we summarize study methodology, present implementation and outcome results, and 
examine the implications of our findings. 


Literacy Design Collaborative Overview 


LDC supports the transition to the Common Core State Standards in English language arts 
by providing flexible module templates that enable middle and high school teachers to integrate 
reading, research, and writing standards into their content-area instruction. End-of-module, 
extended writing tasks provide the heart of the approach. Teachers use fill-in-the-blank templates 
to design a culminating content-focused writing task, which then is used to organize a module of 
instruction. The module is designed to address relevant content in literature, history/social 
studies, or science as well as relevant reading and writing demands aligned with the CCSS. For 
example, the following templates structure end-of-module tasks for students’ argumentative and 
expository writing respectively: 

TASK 1 TEMPLATE (Argumentative/Analysis L1, L2, L3): After researching 

(informational texts) on (content), write an (essay or substitute) that argues 

your position, pro or con, on (content). Support your position with evidence from 

your research. L2. Be sure to acknowledge competing views. L3. Give examples from past or 


current events or issues to illustrate, clarify, and support your position. (Appropriate for: 
social studies, science) 


TASK 11 TEMPLATE (Informational or Explanatory/Definition L1, L2): After researching 


(informational texts) on (content), write a (report or substitute) 
that defines and explains (content). Support your discussion with evidence from 
your research. L2. What implications can you draw? (Appropriate for: ELA, social studies, 


science) 


After deciding on the end-of-module writing task, teachers then use an LDC-specified 
framework (or instructional ladder) to design instructional activities to support students in 
developing the content and requisite literacy skills to successfully complete the culminating task. 
The steps of the ladder include core activities, such as note-taking, identifying evidence to 
support claims, and evaluating contrasting positions, that scaffold student learning and provide 
ongoing opportunities for formative assessment. The final product—instructional ladder plus 


template task—is referred to as an LDC module. 


The Foundation has been exploring a variety of approaches and partners to support LDC 
implementation. The approaches vary in the extent of professional development and coaching 
support that teachers and schools receive and in their focus on individual teachers, or districtwide 
and/or schoolwide implementation. Depending on the district or school context, teachers work 
individually or collaboratively with other teachers and/or specialists to create the modules, which 
typically are subjected to a process of review and refinement. In some settings, all teachers for a 


particular course and grade use common modules. 


As we describe later, teachers in the current study participated in two to three professional 
development sessions during the study year. Study teachers were in almost all cases expected to 
implement at least two modules during the academic year, with one targeting explanation and the 


other focused on argument. 
Evaluation Questions 


At the time of the study, study teachers had had only one or two years of prior experience 
in implementing LDC and were part of the initial trials of LDC with early district implementers. 
At this early phase of LDC development, the study addresses a comprehensive set of evaluation 
questions: 

1. How do teachers implement LDC? 

2. What is the impact of LDC on student learning? 


3. What conditions and contexts, including quality of implementation, influence LDC 
effectiveness? 


In addressing these questions, the study implemented a quasi-experimental design and 
developed and validated new measures of implementation and learning impact, as described in 


the next chapter. 


Chapter 2: Study Methodology 


The study focused on eighth-grade teachers of history/social studies and science and their 
students in both Kentucky and Pennsylvania to study program effects over the 2012-2013 school 
year. Teachers in the study were early implementers of LDC. Study methodology featured a 
strong quasi-experimental design to examine LDC’s effects on students’ state assessment 
performance, coupled with implementation and student outcome measures that were specially 
developed to align well with LDC goals. The implementation measures included logs, teacher 
surveys, and analysis of LDC modules with accompanying student work. Below we provide 
more detail on these elements of the study methodology. 


Study Sample 


Population. The study population was centered in five districts across Kentucky and six 
districts located within one Intermediate Unit region in Pennsylvania. These districts were the 
earliest adopters of LDC, part of the Phase 1 LDC implementation in 2010-2011 and its Phase 2 
expansion in 2011-2012. Within these districts, we sought to include in our sample all Phase 1 
and Phase 2 teachers of eighth-grade history/social studies and science who taught in the 2012- 
2013 school year, for a total of 36 Kentucky teachers and 24 Pennsylvania teachers. Combining 
teachers across states and course subjects was necessary to maximize the available statistical 
power. Eighth grade was selected as a focus because both Kentucky and Pennsylvania administer 
a writing assessment at Grade 8, in addition to standardized measures of reading and language, 
and the Kentucky state assessment also includes an eighth-grade social studies measure. These 
measures were used in the quasi-experimental design (QED) to examine LDC effects on student 


learning. 


The breadth of LDC implementation within schools varied across states, districts, and 
schools. In some sites, all social studies and science teachers in eighth grade participated in LDC, 
while in other sites participation was voluntary. Almost all LDC teachers in the study population 
implemented at least two LDC modules during the study year, 2012-2013. These modules 
typically had been collaboratively developed with at least one partner teacher. The timing for 


implementing these modules was at the discretion of individual teachers. 


Sample for quasi-experimental design. All LDC teachers in the study population and 
their students during the 2012-2013 school year were included in the QED study of LDC effects 
in Kentucky. Using longitudinal student and teacher data from Kentucky’s state database and 
drawing on eighth-grade students in similar courses across the state, we used propensity 
matching techniques to create a comparison sample of students who were equivalent to the LDC 


group in demographics, prior academic performance, and the prior effectiveness of their teachers 


and schools. These techniques and the resulting samples are described in more detail in Chapter 
4 in the context of the QED results. 


Table 1 shows the demographic characteristics of the LDC student sample in Kentucky, 
based on available state data, as well as data on the students’ exposure to LDC. These data 
indicate that the study’s LDC student population was predominantly White, with nearly equal 
representation of males and females. Nearly half of the students qualified for free or reduced 
price lunch. Ten percent of the students were identified as students with disabilities. English 


language learner students were little represented. 


Table 1 


Demographic Characteristics of Kentucky LDC Students: All Eighth-Grade 
Students Taught by LDC Social Studies and/or Science Teachers in the Study 
Population (n= 2,529) 


Demographic characteristic n % 

Ethnicity 

Hispanic 66 3.0 

White 2288 90.0 

Black 65 3.0 

Other (Asian, American Indian, Alaskan Native) 110 4.0 
Qualify for free or reduced price lunch 1177 47.0 
English language learner 11 0.4 
Gender: Female 1241 49.0 
Special education 262 10.0 
LDC Exposure 

LDC in social studies and science 1429 56.5 

LDC in social studies only 827 32.7 

LDC in science only 273 10.8 


Based on the demographics of eighth-grade students statewide who were administered the 
eighth-grade Kentucky state assessment (K-PREP), the study treatment sample appears generally 
similar to the state population, with two apparent exceptions: The study sample has a smaller 
proportion of Black students (3% compared to 10.5%), a larger proportion of White students 
(90% compared to 81%), and a slightly lower representation of students who qualify for free or 
reduced price lunch (47% versus 52%). (See Pearson, 2013, for demographics of the state testing 
population.) 


Over half of Kentucky students in the sample were exposed to LDC in both social studies 
and science, with about a third exposed in social studies only and 10% exposed in science only. 
In Chapter 4, we provide details on how we modeled students’ LDC dosage. 


Table 2 shows the demographics of the Pennsylvania LDC population.' The data indicate 
that, like the rest of the state, the majority of students are White, although statewide the 
proportion is slightly higher than that in the LDC schools—71.7% versus 65.1%. Relative to the 
state population, LDC students are more likely to be Hispanic (8.5% versus 24.7%) and less 
likely to be Black or Asian. LDC students also are somewhat less likely to be economically 
disadvantaged, as measured by free or reduced price lunch status. Presence of English language 
learners is low in both populations. Student performance on the state reading test indicated that 
statewide performance on the seventh-grade reading and math exams in 2011-2012 was slightly 
lower than the performance of the LDC student sample selected for the quasi-experimental 
analysis. The mean seventh-grade reading scale score was 1435 for our sample compared to 
1413 for the state at large, and the mean math scale score was 1529 in our sample compared to 
1500 in the state. Grade 8 mean scale scores for 2012-2013 were not yet published for 
Pennsylvania at the date of publication, so a comparison could not be made on the outcome 
measures to our sample. 


Table 2 


Demographic Characteristics of Pennsylvania LDC Students: All Eighth-Grade 
Students Taught by LDC Social Studies and/or Science Teachers in the Study 
Population (n =1446) 


Demographic characteristic n % 

Ethnicity 

Hispanic 347 24.7 

White 914 65.1 

Black 72 5.1 

Other (Asian, American Indian, Alaskan Native) 66 4.7 
Qualify for free or reduced price lunch 483 33.4 
English language learner 63 4.4 


Sufficient data to support a rigorous quasi-experimental design were not available in 
Pennsylvania, where the study could not gain access to statewide student longitudinal data. 
Instead, feasibility issues limited available data to LDC and demographically similar districts 


‘Complete data on gender were not available. 


within the Intermediate Unit and to student performance on state tests only for the year prior and 
subsequent to the LDC implementation. Because of stakeholder interest, we used these data to 
explore LDC effects. Due to data limitations, these QED analyses are summarized in Appendix F 
and are not discussed further in the body of this report. 


Sample recruitment and completion rates for LDC-only measures. District leaders and 
district-level LDC coordinators in both states fully supported study recruitment, but teacher 
participation in the special measures was totally voluntary. The effective study sample size thus 
varied with the various instruments used in the study. Table 3 and Table 4 display the number of 
study-eligible teachers who were invited to participate in the study, those who agreed to 
participate, and the completion rates for each of the study measures in Kentucky and 
Pennsylvania respectively. Because no special agreement was needed to include teachers in the 
analysis of available state assessment and demographic data, all teachers in the study population 
were included in the study sample. 


In Kentucky, approximately half of the eligible eighth-grade teachers agreed to participate 
in the study and completed the major research activities, including logging about modules, 
submitting module materials, and administering the specially developed student learning 
measure, the Integrated Learning Assessment (ILA). Unfortunately, fewer than half of the 
Kentucky teachers who administered the ILA also returned the one-page opportunity-to-learn 
(OTL) survey that was included with the assessment materials. The teacher survey, which 
demanded a smaller amount of teachers’ time than the other data collection components of the 
study, attracted a larger proportion of eligible participants: three quarters of our Kentucky LDC 
teacher sample completed the survey. 


Table 3 
Study Completion Rates: Kentucky 


LDC teachers (eighth-grade % relative to all % relative to 
history/social studies and science) n eligible LDC teachers _—_ consented teachers 


Teachers eligible for CRESST study and teacher survey 36 -- -- 
Teachers consenting to CRESST study 19 53 -- 


Teachers completing: 


Logs 18 50 95 
Teacher artifacts 18 50 95 
ILA 18 50 95 
OTL survey 8 22 42 
Teachers participating in survey 27 75 -- 


In Pennsylvania, at the request of our partners in the Intermediate Unit, we relaxed our 
eligibility requirements to allow teachers new to LDC to participate in the study. This was a 
strategy to increase buy-in on the part of the small districts participating in the study, and had the 
effect of allowing 10 additional teachers to participate. We did not, however, change the 
eligibility requirements for the separately administered teacher survey, and therefore the samples 
were different for these two separate data collection efforts. With the strong support of the 
Intermediate Unit, recruitment was very successful, with over 90% of targeted teachers 
consenting to participate in the research (see Table 4). The large majority of these teachers 
completed all major research activities, although a smaller majority completed the OTL survey. 
The completion rate for the teacher survey also was quite high. 


Table 4 


Study Completion Rates: Pennsylvania 


LDC teachers (eighth-grade history/social studies % relative to all eligible % relative to 
and science) n LDC teachers consented teachers 
Teachers eligible for CRESST study 24 -- -- 
Teachers consenting to CRESST study 22 92 -- 


Teachers completing: 


Logs 20 83 91 
Teacher artifacts 20 83 91 
ILA 20 83 91 
OTL survey 15 63 68 
Teachers eligible for teacher survey 16 -- -- 
Teachers completing teacher survey 14 88 -- 


Implementation Measures 


Our implementation measures draw on research on instruction and instructional change, 
given that the ultimate goal of the LDC intervention is to align teachers’ instruction to the 
Common Core State Standards. Classroom practice is notoriously impervious to reform (Cuban, 
1984; Lortie, 1975); however, an emerging body of research has documented the relationship 
between student achievement and specific instructional practices that create opportunities to 
learn (see Bryk, Sebring, Allenworth, Luppescu, & Easton, 2010; Rowan & Correnti, 2009; 
Winters & Herman, 2011). Our implementation measures thus focus on classroom instruction, 
while recognizing that multiple factors influence and inhibit teacher innovation and instructional 
change. The measures include web-based teacher logs, collected twice weekly during LDC 
module implementation, collection and analysis of LDC modules, and a teacher survey. In all 
measures, teachers were asked to focus on the same focal class, if they taught more than one 
section of eighth-grade history/social studies or science. 


Web-based teacher log. Study teachers were asked to complete a log twice weekly during 
their implementation for each of two LDC modules, one implemented in the fall and the second 
in the spring. The logs focused on (a) the degree to which instruction generally aligned with the 
structure of the LDC intervention, (b) the degree to which instruction explicitly specified and 
addressed the discrete literacy skills required to complete the summative writing task, and (c) the 


quality and extent of formative assessment practices incorporated into LDC instruction. Each log 


10 


was designed to capture classroom instruction on the particular day the log was completed and 


focused on only one of the teacher’s classes—the same class for all logs. 


The log included opening (gateway) items that asked teachers to specify which component 
of the LDC module they addressed on that particular day (i.e., Preparing for the Task, Reading 
Process, Transition to Writing, Writing Process) and then branched to back-end items for the 
identified component(s), where teachers answered additional questions about component 
instructional objectives and strategies. For example, the Reading Process section asked teachers 
to check all the specific reading skills that were addressed in the day’s instruction. Follow-up 
items asked teachers to identify how they assessed student understanding and/or reading skills 
during the period and how they responded if a student had difficulty with the reading assignment. 
The emphasis on formative assessment aligns with LDC intent, which views the steps in the 
instructional ladder as opportunities for the teacher to track student progress and intervene 


appropriately to support student learning. Logs were analyzed at the teacher level. 


Participating teachers were asked to complete logs twice per week over the course of two 
LDC modules: one implemented in the fall and one in the spring. Almost all teachers logged 
about two different modules, with the exception of one Kentucky teacher who only implemented 
one module during the school year, and one Pennsylvania teacher who failed to submit logs for 
one of her two target modules. On average, Kentucky teachers submitted 11 logs for the two 
modules, and Pennsylvania teachers submitted 10 logs. There was, however, a great deal of 


variation across teachers, with individual participants submitting between four and 15 logs. 


Log data were aggregated by teacher and then summarized across teachers for four groups 
of teachers: Kentucky history/social studies, Kentucky science, Pennsylvania history/social 
studies, and Pennsylvania science. That is, item-by-item mean scores were computed for each 
teacher across all logs that teacher submitted for a given module (Module 1 or Module 2) and 
across the two modules. The computations included teachers’ responses only to those items that 
were associated with the LDC component(s) that the teacher specified for each log. For example, 
log reading component mean scores were based only on responses to logs for which teachers 
reported implementing the reading component of the module. (See Exhibit A1 in Appendix A for 
a copy of the Kentucky/Pennsylvania log.) Group means were then computed across teachers and 
modules. Implementation analyses also considered various composite measures, as described 


further in implementation results below. 


Analysis of LDC modules. Teachers were asked to submit modules and classroom 
artifacts directly into the online survey engine when they completed logging about a given 
module. These artifacts included 


11 


1. a completed template task (often printed from Module Creator, an online tool for 
developing a module, available to many LDC teachers); 


2. copies of all texts used in the module; 


3. one sample of supplemental instructional materials used during the reading component 
and one from the writing component (e.g., graphic organizers, worksheets, lesson 
plans) that each spoke to the specificity of instruction; and 


4. three samples of student work on the template task, marked high, medium, and low. 

Teachers also had an opportunity to submit any additional materials that they believed 
would help us understand their module instructional practice. We followed up with teachers who 
had completed logging but did not submit their materials. As the data in Table 3 and Table 4 
above show, module materials were received from all teachers who completed the logs. Our final 
count of collected materials was 22 Kentucky social studies modules, 13 Kentucky science 
modules, 20 Pennsylvania social studies modules, and 18 Pennsylvania science modules. These 
73 modules represented nearly all of the modules on which teachers logged during the 


implementation study. 


Raters used the specially developed CRESST Assignment Measure rubric to score the 
Kentucky and Pennsylvania modules on nine dimensions of quality. Attending to both content 
and literacy demands, the dimensions address the quality of the central writing task and the texts 


it draws on, the quality of the instructional ladder, and overall module coherence: 
e Dimension 1: Effective Writing Task 


e Dimension 2: Alignment to the CCSS and Local and State Literacy and Content 
Standards 


e Dimension 3: Text Alignment 

e Dimension 4: Text Appropriateness 

e Dimension 5: Text Rigor 

e Dimension 6: Fidelity to LDC Module Instruction 

e Dimension 7: Quality Instructional Strategies 

e Dimension 8: Coherence and Clarity of Module 

e Dimension 9: Overall Impression 

Each dimension was scored on a 1—5 scale, where a score of 1 indicated poor quality, a 
score of 3 indicated the quality was moderately realized, and a score of 5 indicated that the 


quality of the dimension was fully realized (see Exhibit A2 in Appendix A for the scoring 


rubric). 
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Subject matter teachers in history/social studies and science were recruited as scorers and 
received special training to ensure they could consistently apply the rubric to the collected 
modules. The training provided detailed rubrics for each dimension, exemplified by anchor 
papers (i.e., module components) demonstrating each score value, and multiple opportunities for 
practice and feedback on rubric application. Scorers established their consistency before 
embarking on scoring and consistency was checked throughout the scoring process. The 
measurement quality of the resulting scores was established through generalizability, factor 
analysis, and decision study methodologies and is reported as a separate paper (see Reisman, 
Herman, Luskin, and Epstein, 2013, in Appendix B, which describes the measures, including 
development, piloting, scoring, generalizability and dependability studies, and results). Raters 
generally found the scoring dimensions intuitive and well aligned with the available artifact data. 
Both the social studies and science analyses revealed low rater variance across scoring 
dimensions (between 0% and 14% of total variation depending on the dimension and subject) 
and high teacher and/or teacher by module variation (between 28% and 72% depending on the 
dimension and subject), suggesting that the scores were capturing real differences in module 
implementation across teachers. Moreover, based on factor analyses, all nine dimensions loaded 
on a single factor for both subjects, making the case that the CRESST Assignment Measure 
effectively measures a coherent trait that might be understood to be LDC implementation, or 
perhaps more generally, instructional quality in the integration of literacy and content. In Chapter 
3, we use dimension scores to provide descriptive results on quality of implementation and 
overall mean scores to examine the relationship between module implementation and student 
performance. 


Teacher surveys. CRESST collaborated with Research for Action (RFA) on the design of 
a 2013 implementation and scale-up survey for teachers. The survey included a section on 
module implementation with items designed to mirror the intent of the CRESST log measure 


items. These survey items queried 
e relative time spent on the various module components; 
e relative emphasis given to specific reading and writing skills; 
e use of formative assessment and strategies for providing feedback; and 
e perceptions of LDC impact. 
Further, we drew on RFA survey variables as context and possible moderators of LDC 


implementation and impact—for example, experience using LDC, attitudes regarding literacy 


instruction, extent of professional development, leadership support, and collaboration. 
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Descriptive statistics were computed at the teacher level. (See Exhibit A3 in Appendix A for a 
copy of the LDC teacher survey.) 


Student Outcome Measures 


Student outcome measures for the study include state assessment data and CRESST- 
developed Integrated Literacy Assessments (ILAs). Student demographic information also was 


secured with the available state assessment data. 


State assessment data: Kentucky. The study used data from the Kentucky Performance 
Rating for Educational Progress (K-PREP) to measure students’ 2012-2013 performance in 
English language arts, writing, and social studies. K-PREP contains both multiple choice and 
short constructed response items in a blended model of criterion- and norm-referenced testing. 
The writing assessment features an on-demand writing sample in which students create an essay 


in response to reading a single passage. 


Reported reliability for the eighth-grade reading and social studies tests are .87 and .90 
respectively (Pearson, 2013). 


State assessment data: Pennsylvania. The study used available data on student 
performance on the Pennsylvania System of School Assessment (PSSA) in reading. Data were 
requested on eighth-grade LDC and comparison students for 2012—2013 and for these students’ 
prior performance as seventh-graders in 2011—2012. Data were provided by local districts within 
the Intermediate Unit. 


The PSSA reading tests are composed predominantly of multiple choice items, but also 
include several constructed response items. Reported reliability based on coefficient alpha is .91 
for the 2012-2013 eighth-grade reading test, .89 for the seventh-grade reading test in 2011-2012, 
and .93 for the seventh-grade math test in 2011-2012 (Data Recognition Corporation, 2012, and 
Data Recognition Corporation, 2013; see _http://www.portal.state.pa.us/portal/server.pt/ 
community/pssa_technical_reports/7447). 


The original study design also included students’ eighth-grade PSSA writing scores, which 
are based on a direct writing assessment. Unfortunately, however, because of substantial missing 
data, these scores could not be included in study analyses. Moreover, limited demographic data 


were available for all students in our sample. 


Integrated Literacy Assessment (ILA). The CRESST ILAs are designed to measure both 
students’ literacy development relative to the CCSS in English language arts and the depth of 
students’ content understanding in literature, history/social studies, or science. Across content 


areas, the two-day ILAs feature a consistent structure that roughly mirrors components of LDC: 
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On Day 1, students read several texts that typify those encountered in the discipline and address 
an important content principle or theme and respond to selected and constructed response 
reading comprehension and analysis questions about each text. The questions are aligned with 
the CCSS in ELA. On Day 2, students respond to an essay prompt that, consistent with the 
CCSS, asks them to synthesize what they know with what they have read to produce an 
evidence-based, extended explanation or argument responding to a content-related problem. 
Student essays were scored by trained raters using a generalized, analytic scoring rubric that was 
customized for each prompt. Table 5 and Table 6 show the dimensions addressed by the writing 


task rubric and an example of the score values for each one of the dimensions. 


Table 5 
ILA Scoring Rubric for Final Writing Task 


Dimension Name Description 

A Content This is a measure of overall how well the student has demonstrated that they 
understanding understand the materials and the topic in their essay. 

B Rhetorical Argument: establishes a claim, acknowledging alternate or opposing claims, 


structure/quality and supports it consistently with relevant evidence and logical reasons. 


Explanation: establishes a thesis; previews the main points; and thoroughly 
develops the topic with well-chosen information, examples, and analysis. 


C Organization Consistent focus, logical progression of ideas, and structure appropriate for the 
task. 
D Reference/support This is a measure of how well statements in the essay are supported by 
with text references to text details. A text detail is a quotation, paraphrase, or any other 


reference to information and ideas in the texts provided. 


E Grammar and The essay is written with a command of standard English conventions: proper 
conventions English usage and control of grammar, appropriate tone, paragraph, and 
sentence structure. 


Table 6 


Scoring Dimension Example for Rhetorical Structure/Quality 


Description Score 


Important elements of the argument are clearly and thoroughly described and articulated. 
Elements of the argument are clearly described. 


There is an attempt to describe some elements of the argument. 


PN WwW 


Elements of the argument are not described, or the descriptions are unclear. 
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ILA content foci. The ILA design seeks to respond to the challenge of disentangling the 
background information that students bring to the assessment from the knowledge that they 
gather from reading the actual texts in the assessment (Klein, 1983). Assessment designers and 
evaluators must take care to not unfairly privilege students whose teachers spent considerable 
time on a topic over those whose teachers’ coverage was more cursory. We addressed this 
potential confound in the ILAs in two ways: first, we selected topics that students should have 
covered in their recent curriculum, or that were closely related to topics they had covered; 
second, we included relevant background knowledge in the actual exam so that even students 
with virtually no familiarity in the topic could orient themselves to the substance of the texts and 
write meaningfully about them (Baker, 1994). 


The history/social studies ILA used in the study focuses on Reconstruction following the 
Civil War. The assessment includes background information in the form of a short summary 
describing Reconstruction and a timeline of key events. Each of the three documents in the 
assessment includes a headnote with key background information about the author and the 
context. The three documents—a speech by Frederick Douglass, an excerpt from South 
Carolina’s Black Codes, and a freed person’s testimony of KKK violence—vividly describe a 
wide range of challenges faced by African Americans in the Reconstruction-era South. The 
writing task was designed as an argument task; students were asked to argue that African 
Americans were not actually free during the Reconstruction era using evidence from the three 
documents. Although Reconstruction is not explicitly part of the eighth-grade standards in 
Kentucky, students learn about U.S. history through the Civil War, right up to Reconstruction. 
Given the extensive background knowledge included in the assessment, we believed it would 


effectively gauge students’ ability to learn from text and write effectively about it. 


The science ILA focused on science inquiry skills in the context of Darwin’s theory of 
evolution. Students were asked to read and respond to questions about three documents: one a 
description of “fitness” and how the term is used when discussing evolution; a second which was 
a diagram related to the process of natural selection and how it is an agent of evolution and leads 
to change over time; and the third a short description of an actual scientific study investigating 
mate choice by females in a group of birds called widowbirds. The final extended essay task 
asked students to create an argument to refute a given (incorrect) assertion. The content and 
intellectual demands of the task aligned with academic standards in both states. Kentucky’s 
science standards at the time (prior to their 2013 adoption of the Next Generation Science 
Standards) used “diversity and unity” as one of the eight major organizing themes for its Grade 8 
expectations, and specified that students develop their scientific thinking and reasoning through 
using scientific methods to solve real-life problems by identifying, analyzing, and using patterns 
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to understand present events and predict future events (see Kentucky Core Academic Content 
Standards for eighth-grade science, http://education.ky.gov/curriculum/docs/Documents/POS 
with CCS for public review.pdf). Similarly, Pennsylvania’s eighth-grade science standards 
require that students be able to explain theory, use evidence to support arguments, and 
understand and be able to explain basic concepts of adaptation and survival (see Pennsylvania 
eighth-grade science standards, http://www.pdesas.org/standard/views). 


Copies of the science and history/social studies ILAs administered in the study, with 
accompanying rubrics, can be found in Appendix A: Exhibits A4, A5, and A6. 


ILA administration and scoring. [LAs were administered at the end of the 2012-2013 
school years in LDC classrooms only. Teachers were sent test materials, including student test 
forms and directions for administration, and asked to administer the assessment to the one class 
that had been the target of their logs. The ILAs were administered over two class periods. On 
Day 1, students read the texts and responded to selected and constructed response reading 


comprehension and content analysis questions. 


Six secondary content teachers were recruited for the essay scoring, including three 
history/social studies teachers and three science teachers. Four of the six had been involved in 
prior ILA scoring projects. Scorer training provided orientation to each prompt and rubric 
dimensions for scoring it. After reviewing anchor papers (prescored criterion papers exemplified 
each dimension and score point) teachers practiced scoring and received feedback on their use of 
the rubric. Teachers had to qualify for actual scoring by demonstrating at least 80% agreement 
on qualifying sets of expert-scored papers. 


One social studies rater and one science rater failed to achieve sufficient reliability after 
multiple training sessions and thus did not participate in the essay scoring, leaving two raters for 
each subject area. Essays were divided among these raters and a proportion of papers were 
double-scored to document reliability. In addition, check papers were threaded throughout the 
scoring process and raters who veered from expected ratings were provided feedback and 
additional training, as necessary. Specific rubrics were developed to score short answer 
questions. It was possible for students to be awarded partial credit on a select number of short 


answer questions. 


ILA reliability. Table 7 and Table 8 display rater reliability in the scoring of the Evolution 
and Reconstruction ILAs. As Table 7 shows, raters achieved exact agreement of 70% or more for 
four of the five dimensions and just missed for the fifth dimension (organization, 69% exact 


agreement). For the Reconstruction ILAs, two dimensions fell below 70% exact agreement: 


17 


rhetorical structure and quality at 65% and grammar and conventions at 55%. However, there 


was Virtually 100% agreement plus or minus one score point. 


Table 7 
Interrater Reliability for Double-Scored Evolution ILA Essays (n = 54) 


Dimension % exact agreement % agreement within one point 
Content understanding 70 96 
Rhetorical structure/quality 78 100 
Organization 69 91 
Reference/support with text 76 100 
Grammar and conventions 74 100 


Table 8 


Interrater Reliability for Double-Scored Reconstruction ILA Essays (n = 80) 


Dimension % exact agreement % agreement within one point 
Content understanding 71 100 
Rhetorical structure/quality 65 100 
Organization 70 100 
Reference/support with text 73 100 
Grammar and conventions 55 100 


Table 9 displays reliability statistics for the two assessments. To maximize sample size for 
our reliability tests, we pooled assessments across both states. As can be seen in the table, 


reliability across writing dimensions and reading items was quite high for both assessments. 


Table 9 


Reliability of Evolution and Reconstruction ILAs 


Cronbach’s alpha 


Assessment Component Number of students Number of items/dimensions (reliability) 
Evolution Reading 388 13 .717 
Evolution Writing 335 5 823 
Reconstruction Reading 449 16 724 
Reconstruction Writing 458 5 885 
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Opportunity to learn ILA content. Despite the care with which ILA topics were selected 
to align with eighth-grade content standards in both states, responses to the study’s post- 
intervention opportunity-to-learn survey shows some mismatches. Teachers completed this short, 
one-page survey at the time that they administered the ILAs. In addition to asking about details 
of ILA administration (i.e., the administration date and amount of time students spent completing 
them), the survey asked teachers to summarize their LDC instruction (number of modules taught 
in 2012-2013 and topics covered). The survey also asked teachers to report on the degree of 
emphasis they placed in their 2012-2013 instruction on the content areas covered by the ILAs: 
for LDC history/social studies teachers, the Reconstruction period in American history; and for 
LDC science teachers, the scientific theory of evolution. (See Exhibits A7 and A8 in Appendix A 
for copies of the Evolution and Reconstruction OTL surveys.) 


Unfortunately, as noted earlier, return rates were not high for this survey, particularly in 
Kentucky. Just three of seven Kentucky science teachers returned the short survey along with 
their students’ assessments, while eight of 10 Pennsylvania science teachers returned the survey. 
In social studies, four of 11 Kentucky teachers and seven of 10 Pennsylvania teachers returned 


the survey. 


Responses in Table 10 and Table 11 show surprisingly low coverage of ILA topics, 
particularly for science. Only one of three responding Kentucky science teachers reported 
placing any emphasis on evolution, and this emphasis was slight. In Pennsylvania, none of the 
eight science teachers reported any emphasis on evolution in their classrooms. In Kentucky, half 
of social studies teachers reported placing no emphasis on Reconstruction and half reported 
placing slight emphasis. Pennsylvania teachers on average placed more emphasis on 
Reconstruction, with the majority placing at least some emphasis on the subject (recall that 
history standards for eighth grade went through the Civil War, so it shouldn’t be surprising that 
some teachers reported not covering Reconstruction). 


These data provide important context for interpreting ILA results. At the same time, 
however, the texts that students were asked to read during the ILA provide sufficient information 
for them to respond to the essay prompt. 
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Table 10 


Science Teachers Reporting Level of Emphasis Placed on Evolution in Their Classes 


No emphasis Slight emphasis Moderate emphasis Sustained emphasis 
State n % n % n % n % 
Kentucky 2 66.7 1 33.3 0 0.0 0 0.0 
Pennsylvania 8 100.0 0 0.0 0 0.0 0 0.0 


Table 11 


Social Studies Teachers Reporting Level of Emphasis Placed on Reconstruction in Their Classes 


No emphasis Slight emphasis Moderate emphasis Sustained emphasis 
State n % n % n % n % 
Kentucky 2 50.0 2 50.0 0 0.0 0 0.0 
Pennsylvania 2 28.6 2 28.6 3 42.9 0 0.0 
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Chapter 3: LDC Implementation 


In this chapter, we present descriptive findings from our implementation measures, 
including teacher logs, surveys, and analysis of LDC modules. Results were analyzed separately 
by state and subject area to explore potential implementation differences, but in the interest of 
space, the tabled data are provided in two appendices (Appendix C for survey results and 
Appendix D for log results). In reviewing these findings, it is important to keep in mind the small 
sample sizes and that while log, survey, and module samples overlap, they are not fully the same. 


For example, some teachers completed the survey but not the log and vice versa. 
Teacher Background 


Teachers’ background, prior experience, and attitudes about literacy instruction, gleaned 
from teacher survey responses, provide important context for the implementation findings. 
Survey responses indicate a wide range of experience among participating Kentucky and 
Pennsylvania teachers. On average, Kentucky teachers and Pennsylvania science teachers had 
between 12 and 14 years of experience, with a range of three to 32 years of prior experience. 
Pennsylvania social studies teachers were somewhat less experienced, with a mean of eight years 
(see Table C1 in Appendix C). 


Typical participating teachers in both states reported spending the majority of their 
teaching careers in the same district and at the same school. The majority of teachers in both 
states had special education students, students reading or writing below grade level, and students 
with advanced literacy skills in their current classrooms. Kentucky teachers had less experience 


than Pennsylvania teachers in teaching English language learners (see Table C2). 


Teachers’ reports on whether they were required to use LDC varied considerably across 
states and subjects. Nearly all Kentucky science teachers reported that LDC use was required, as 
did about two thirds of social studies teachers in both states. In contrast, half of responding 
Pennsylvania science teachers reported that their LDC participation was voluntary (see Table 
C3). 


There was a fair amount of variation in teacher experience with developing and teaching 
LDC modules. All responding teachers reported that they had developed at least one LDC 
module during the 2012-2013 school year with the average teacher reporting a role in 
developing just under two modules. Based on the number of modules taught in 2011-2012 and 
2012-2013, Pennsylvania teachers and Kentucky social studies teachers on average taught about 
four modules over the two school years. Kentucky science teachers, however, only taught an 


average of about 2.5 modules during this period. Note as well that although all study teachers 
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were at least trained in LDC by the 2011-2012 year, the range of responses indicate that at least 
some had not implemented a module until 2012-2013, the study year (see Table C4). 


Generally, participating teachers in Kentucky and Pennsylvania agreed that content area 
teachers share responsibility for building students’ literacy skills, and that writing can help 
students develop deeper conceptual understanding. Some respondents, particularly in Kentucky, 
however, were concemed that content area teachers do not have sufficient time to teach reading 


and writing (see Table C5). 
Log Findings 


The log data provide information on the forms of activities in which students were engaged 
during module implementation, the specific reading and/or writing activities in which students 
participated, and teachers’ use of formative assessment. Note that there was wide variation 
across teachers, so the means reported below must be interpreted with caution. Results are 
reported by module and overall for each of the four categories of teachers (Kentucky and 


Pennsylvania social studies and science). 


Across both states and subjects, teachers reported that by far, the most frequently used form 
of instruction was independent reading and/or writing, constituting half or more of classroom 
LDC time. Explicit strategy instruction—directly supporting student skill development—was 
relatively infrequent, making up on average less than 10% of classroom time; mini-lessons were 


also infrequent (see Tables D1 through D4 in Appendix D). 


During the LDC reading component, teachers uniformly reported that independent reading 
and research, note-taking and annotation, summarizing information, and vocabulary were likely 
to be at least touched on briefly or a major focus of instruction. Social studies teachers in both 
Kentucky and Pennsylvania also reported some degree of focus on critical reading skills, such as 
drawing conclusions from text, citing textual evidence to support claims, and evaluating 
strengths and weaknesses of evidence. Building these skills seemed to be less of a focus in 
science classrooms in either Kentucky or Pennsylvania. Across states and subjects, critical 
analysis, such as comparing arguments, examining authors’ perspectives and/or bias, 
distinguishing fact from opinion, and analyzing text structure were less emphasized, but were at 


least touched on according to most teachers’ reports (see Tables D5 through D8). 


Responses to log items on formative assessment strategies teachers used during the reading 
process component also illuminated differences between social studies and science teachers. 
While both social studies and science teachers reported frequently circulating and reviewing 
student work, social studies teachers were more likely to use a wider variety of strategies, such as 


listening as students had discussions about text and engaging students in questioning. If teachers 
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discovered misunderstandings, the most frequently cited strategy to address the 
misunderstanding was to conduct a one-on-one conference with a student. There also was some 
evidence of teachers using other approaches such as stopping the class and modeling a strategy, 
offering hints and suggestions, or simply giving students more time to self-correct (see Tables 
D9 through D16). 


During the writing component of LDC modules, teachers across both states and subjects 
reported considerable attention to a number of writing skills, including text structure, how to 
write different types of paragraphs (introduction, body, conclusion), and incorporating quotes 
and evidence from texts. Formulating a thesis statement seemed to be a stronger focus in 
Kentucky than in Pennsylvania (see Tables D17 through D20). 


The most commonly cited strategies for assessing student understanding during the writing 
process component were observing and reviewing student work, and reviewing students’ rough 
drafts of the writing task. Teachers tended to use similar strategies to respond to 
misunderstanding as they did during the reading process component, including holding one-on- 
one conferences with students, offering hints or suggestions, or allowing more time for students 
to self-correct (see Tables D21 through D28). 


Teacher Survey Responses 


Below we focus on teacher survey responses to items aligned with the log foci, followed by 
results on variables likely to influence teachers’ LDC implementation. The latter includes 
responses to a series of additional questions on teacher efficacy, school and district support for 
LDC, professional development, teacher collaboration, and perceptions of the effectiveness and 
impact of the initiative. As with the log data, survey responses show substantial teacher 


variation. 


LDC implementation. The survey queried teachers about the relative time they spent on 
each of the four components of an LDC module: Introduction/orientation, reading, transition to 
writing, and writing. Responses suggest that Kentucky teachers spent relatively the most time on 
the writing process component, which accounted for about 36% of the LDC time, followed by 
the reading process component, which drew about a quarter of the module instructional time. 
Kentucky science teachers spent relatively more time introducing the module, while Kentucky 
social studies teachers focused more of their time on transitioning to writing. In Pennsylvania, 
teachers reported allocating roughly equal time to reading and writing, with about a third of the 
total module time devoted to each (see Table C6). 


Similar to log responses, Kentucky and Pennsylvania teachers reported giving at least some 


attention to a range of reading skills and strategies during the reading component. The vast 
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majority of teachers reported giving at least some attention to all skill areas queried on the 
survey, with the exceptions being analysis of rhetorical devices, which drew little attention from 
all teachers, and examining authors’ perspective, which was not typically a focus of science 


teachers. 


There were indications that patterns of strong emphasis varied between the two states and 
subjects. For example, while large majorities of Pennsylvania teachers reported giving heavy 
emphasis to summarizing important points of reading, note-taking, and independent reading, 
these skills drew relatively less attention in Kentucky. In contrast, Kentucky social studies and 
science teachers particularly emphasized one skill area: drawing conclusions from evidence. In 
general, science teachers placed great emphasis on a smaller set of skills, including independent 
reading, summarizing important points, note-taking, and drawing conclusions from text (see 
Tables C7 through C10). 


Turning to skill and strategy emphases during the writing component, survey responses 
indicate that teachers also gave substantial attention to a wide range of writing skills and 
strategies—a majority in each group gave at least some attention to nearly all of the skills 
queried by the survey. There was considerable variation across states and subjects, however, in 
the writing skills on which teachers placed strong emphasis. Kentucky teachers gave a great deal 
of emphasis to a larger group of writing skills than Pennsylvania teachers. Incorporating quotes 
and evidence seemed to be more important to science teachers (the only skill over half of 
Pennsylvania science teachers placed great emphasis on). Formulating counterarguments, 
particularly in Pennsylvania classrooms, drew little attention, as did using transitional words and 
phrases (see Tables C11 through C14). 


Kentucky and Pennsylvania teachers further reported using a variety of strategies for 
assessing student learning during the course of instruction. The most common approaches across 
states and subjects were circulating to review student notes and work, asking students oral 
questions, reviewing student rough drafts, and grading student work. Peer-oriented strategies 
such as listening as students discussed reading or writing with peers and asking students to 
provide feedback to each other, were more heavily emphasized in Pennsylvania than Kentucky 
(see Tables C15 through C18). 


Similarly, teachers reported using a wide range of strategies to respond to student 
misunderstandings observed during the course of instruction. However, the degree of emphasis 
on different strategies varied by state and subject. Holding one-on-one conferences with students 
and asking peers to provide feedback were more popular strategies in Pennsylvania. Social 


studies teachers more frequently than science teachers reported giving students more time to self- 
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correct. Grading student work was cited as a frequently used strategy by all four groups of 
teachers. Respondents tended not to use reteaching or reviewing the skill in later lessons as major 
strategies. In addition, the majority rarely or never responded with grammar exercises when they 
noticed problems in student work (see Tables C19 through C22). 


Implementation support. Most teachers in both states and subjects reported that district 
leadership supported the LDC framework, although Pennsylvania social studies teachers and 
Kentucky science teachers were somewhat less likely to agree that district administrators 
understood the initiative. There was considerable variation across teachers in their reports on 
whether their school administrators supported LDC. Nevertheless, teachers in all groups tended 
to agree that school administrators understood the initiative, encouraged teachers to participate in 


it, and prioritized formative assessment. 


Teachers, particularly in science, were considerably less likely to report that they received 
feedback about their LDC instruction from school administrators. Science teachers also were less 
likely to feel that school leaders communicated how LDC was aligned with other initiatives. 
Differences between science and social studies also emerged in reports on visits to classrooms by 
district leaders, school leaders, and colleagues during LDC module instruction. Just one fifth of 
Kentucky science teachers and one third of Pennsylvania science teachers received a classroom 
visit from a district or network LDC project lead, compared to 43% of social studies teachers in 
each state. Likewise one third of science teachers in each subject received visits by principals 
during LDC instruction, while half of Kentucky social studies teachers and nearly all of 
Pennsylvania social studies teachers reported principal visits. The pattern also held for visits 
from instructional coaches, department heads, and teacher colleagues (see Tables C23 through 
C27). 


A gap between science and social studies teachers also is apparent in professional 
development participation; over 85% of social studies teachers participated in formal 
professional development for LDC compared to only half of science teachers. For those teachers 
that did participate, the number of sessions was on average two to three but ranged between one 
and six. Professional development took place in a wide variety of settings, and there was some 
variation across groups. For example, cross-district meetings not surprisingly were more 
common in Pennsylvania (where the Intermediate Unit plays a leadership role). Teachers who 
responded generally found the professional development in all settings to be effective. 
Professional development covered a wide variety of topics, but teachers reported relatively little 
attention on implementing modules with special needs students, including English language 
learners, special education students, and students with either high or low literacy levels (see 
Tables C28 through C37). 
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Turning to teacher collaboration, there was considerable variation across states and subjects 
in the frequency of collaboration. While over 85% of Kentucky social studies teachers, and over 
half of Kentucky science teachers and Pennsylvania social studies teachers reported regularly 
scheduled common planning time to discuss LDC, only one sixth of Pennsylvania science 
teachers did so. Reports on the frequency of formal and informal teacher collaboration varied 
quite a bit across teachers, states, and subjects. Scheduled meetings on LDC occurred at least 
every semester and often more frequently. Informal discussions were not surprisingly more 
frequent. Teachers generally found collaboration with their colleagues to be helpful in a variety 
of ways, including especially the development, implementation, and revision of the LDC 
modules. Kentucky science teachers seemed to find collaboration somewhat less helpful (see 
Tables C38 through C46). 


Perhaps as a consequence of such collaboration and other support, respondents generally 
seemed confident in their ability to implement their modules. Some teachers, however, did 
appear to have concerns about how to use the instructional ladder. In particular, science teachers 
were more likely to report certain barriers to teaching LDC than social studies teachers. For 
example, science teachers struggled more with locating content-rich reading materials at an 
appropriate reading level, and with finding the time to give feedback on student writing and to 
develop modules. Science teachers were also less likely to feel they had sufficient time to 
prepare to teach modules (see Tables C47 through C51). 


Attitudes about LDC efficacy. Teachers generally found LDC to be a helpful and 
effective tool. A majority of teachers in both subjects and states reported that LDC was helpful in 
meeting a wide variety of instructional goals, including implementing the CCSS, teaching 
literacy in content area classes, assessing their students’ literacy strengths and weaknesses, and 
increasing the rigor of writing assessments. Smaller percentages of teachers reported that LDC 
was helpful in better engaging students. Similarly, across states and subjects teachers tended to 
agree at least somewhat that LDC was an effective tool in improving students’ literacy, 
promoting formative assessment, and integrating literacy into secondary and content area 
classrooms (see Tables C52 through C59). 


Despite general support for the initiative, a third or fewer teachers in each group reported 
that students were more engaged during LDC implementation than during non-LDC time. And in 
Kentucky, about a quarter of teachers felt their students were less engaged during LDC than their 
regular instruction (see Table C60). 


The data provide some indication that students are struggling to meet the demands of LDC. 
Although a majority of teachers reported that students experienced at least some success on the 
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LDC final writing task, reading mini-tasks, and writing mini-tasks, fewer than half of teachers 
reported that students had a great deal of success on these tasks. Kentucky social studies teachers 
in particular were less confident about their students’ success on the tasks, with about a third 
reporting students had little success on the writing mini-tasks and the final writing task, and half 
reporting students had little success on the reading mini-tasks. These results suggest that 
Kentucky and Pennsylvania teachers and their students may need help to increase the 
productivity of the reading and writing mini-tasks, as well as success on the final writing task 
(see Table C61). 


However, although there was variation across respondents, teachers tended to agree at least 
somewhat that LDC had resulted in higher quality student writing and supported students’ 
college readiness. Reflecting on their most recent module, most teachers in each of the four 
groups felt that a majority of their students had improved both their understanding of content and 
their literacy skills (see Tables C62 and C63). 


LDC Module Analysis 


As noted earlier, LDC modules and associated student assignments and work were 
collected as part of the log process and were scored on nine dimensions of quality. All teachers 
save one submitted two modules. In the absence of clear score differences, the results below 
combine scores across the two modules and are presented below by subject and state. Social 
studies and science modules were analyzed separately because raters exclusively scored modules 
in their subject area and any differences in subject matter findings may be the result of scorer 
differences. Further, given the small sample sizes and lack of representativeness, any observed 


differences lack generalizability and must be treated as tentative. 


Social studies module results. Table 12 presents descriptive statistics for social studies 
modules by state. The data indicate that module scores generally are similar across the two 
states. Perhaps the biggest distinction is that teachers in Pennsylvania scored considerably higher 
on fidelity to LDC module instruction (D6). The higher score on this dimension likely reflects 
the fact that all teachers from Pennsylvania submitted modules designed on the online LDC 
platform, Module Creator. Approximately half of Kentucky social studies modules were 
submitted using an older paper template for LDC or without any template. If it was difficult for 
raters to identify the four skill clusters—Preparing for the Task, Reading Process, Transition to 
Writing, and Writing Process—the module could not receive a score of 3 on D6. However it is 
important to note that a module could eam a 3 on D6 by simply defaulting to mini-task options 


automatically provided in Module Creator for each skill cluster. In other words, the average 
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score of 3.35 in D6 for Pennsylvania social studies modules does not necessarily suggest that 
those teachers elaborated or expanded on the default options provided for instruction. 


Of the other dimensions, the effectiveness of the writing task and the quality of text 
alignment, appropriateness, and rigor received relatively the highest ratings, indicating at least 
moderate quality. The quality of instructional strategies, module coherence, and raters’ overall 
quality judgments received somewhat lower scores. Mean scores across all dimensions are 
essentially the same for Pennsylvania and Kentucky, 3.03 and 2.99 respectively (not shown in 
the table), indicating moderate quality. As with log and survey data, however, the module data 


also show substantial variation by teacher. 


Table 12 


Descriptive Statistics for Social Studies Modules by Dimension and State (n = 40) 


Pennsylvania (n = 18) Kentucky (n = 22) 

Dimension M SD M SD 
Effective writing task 3.38 0.95 3.38 1.26 
Alignment to literacy and content standards 27. 0.99 2.10 1.19 
Text alignment 3.36 1.09 3.48 1.28 
Text appropriateness 3.07 0.86 3.36 1.03 
Text rigor 3.02 1.03 3.58 1.25 
Fidelity to LDC module instruction 3.35 0.64 2.70 1.33 
Quality instructional strategies 2.80 0.87 2.86 1.13 
Coherence and clarity of module 2.80 0.85 2.80 1.31 
Overall impression 2.76 0.90 2.68 1.04 


Note. Ratings are on a 1-5 point scale, where a score of 1 indicates that a dimension is not in evidence, 3 indicates 
that quality was moderately realized, and 5 indicates that quality is fully realized. 


Science module results. Table 13 presents descriptive statistics for science module ratings 
by state. Results show a general advantage for Kentucky, particularly for fidelity to LDC module 
instruction, and again largely explained by Kentucky teachers’ uniform use of Module Creator. 
Although the difference in the means across dimensions may appear substantial—3.44 for 
Kentucky and 3.05 for Pennsylvania (not shown in the table)—these do not rise to measurement 
significance. Given the high variability in scores, the lack of sample representativeness, and the 
possibility of preexisting differences between teachers, we caution the reader to avoid drawing 


inferences on how modules’ quality may have varied between the two states. 
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Across all dimensions, Kentucky modules averaged moderate quality or higher. Relative 
strengths for the Pennsylvania modules included alignment to standards, text alignment and 


appropriateness, and fidelity to LDC, which were rated as at least moderate in quality. 


Table 13 


Descriptive Statistics for Science Modules by Dimension and State (n = 29) 


Pennsylvania (n = 14) Kentucky (n = 15) 
Dimension M SD M SD 
Effective writing task 2.80 1.15 3.27 1.14 
Alignment to literacy and content standards 3.18 1.11 3.21 1.24 
Text alignment 3.39 1.10 3.62 1.27 
Text appropriateness 3.10 1.14 3.69 1.04 
Text rigor 2.80 1.08 3.25 1.36 
Fidelity to LDC module instruction 3.59 0.79 4.00 1.03 
Quality instructional strategies 2.92 1.00 3.19 1.24 
Coherence and clarity of module 2.92 1.10 3.46 1.23 
Overall impression 2.76 0.99 3.25 1.19 


Note. Ratings are on 1—5 point scale, where a score of 1 indicates that a dimension is not in evidence, 3 indicates that 
quality was moderately realized, and 5 indicates that quality is fully realized. 


Summary of Implementation Data 


Results from teacher logs, surveys, and analysis of teacher-created modules and student 
work provide at least one consistent finding: Across all sources, the data show substantial 
variation across teachers in all aspects of LDC implementation, from how teachers allocated 
instructional time across the various components of LDC, to the primary organizational forms 
teachers used for instruction, the reading and writing skills they most emphasized in LDC 
instruction, and the specific strategies they used to formatively assess and provide students’ 
feedback on their learning. The quality of LDC modules also varied substantially across teachers. 


With this variation as a caveat, the findings provide a portrait of by whom, how, and with 
what support LDC was implemented by the study sample, as well as participating teachers’ 


impressions of effectiveness. 


Who. Survey results indicate that study teachers were generally highly experienced and 
stable in their positions, having spent most of their careers in the same districts and schools. 
Most of the study teachers had one to two years of experience beyond their initial training in 
implementing LDC. All of the teachers had participated in the development of at least one LDC 
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module and the majority had developed two or more modules. Although most of the teachers 
were required to participate in LDC, rather than having volunteered to do so, they felt committed 
as content teachers to help develop their students’ literacy skills. 


How. Log, survey, and module analysis results indicate that teachers followed the LDC 
framework. As indicated by the logs and surveys, students were engaged in independent reading 
and writing during the majority of LDC instructional time. While teachers tended to at least 
touch upon a wide variety of reading and writing skills during this time, they spent relatively 
little time in direct strategy instruction or in delivering mini-lessons. Note-taking and 
summarizing appeared to be relatively frequent student activities during independent reading, 
and for social studies classrooms, critical reading skills such as citing and evaluating evidence 
and using it to draw conclusions also were in evidence. There was little attention to critical 
analysis and synthesis skills, such as differentiating fact and opinion, comparing arguments, or 
analyzing authors’ perspectives. In writing, teachers also reported some attention to a wide range 
of skills but with a relative emphasis on elements of structure. Across both reading and writing, 
teachers reported engaging in frequent formative assessment, involving multiple strategies for 


monitoring student learning and for responding to student misunderstandings as they occurred. 


Analyses of teacher-developed modules provide a window into the quality with which LDC 
is being implemented. Fidelity to the LDC framework was judged a relative strength in the 
ratings, and ratings across most of the nine dimensions examined either approach or achieve 
moderate levels of quality. Results, however, also suggest room for improvement, which is to be 
expected given participating content teachers’ experience levels with LDC and with teaching 
literacy. 


With what support. Survey responses indicated that teachers felt their district leadership 
supported the LDC intervention, but school-level support was less consistent across the sample. 
All teachers participated in professional development and found it beneficial. Teachers found 
their colleagues collaborative, although formal time for planning and collaboration was uneven 
across the sample. Nonetheless, teachers reported that collaboration with their peers was very 
helpful in implementing LDC. Science teachers appeared to be less involved in professional 


development and collaboration than were their history/social studies peers. 


Attitudes toward LDC. Teachers reported that they found LDC a helpful and effective 
tool in meeting a variety of goals, including implementing the Common Core State Standards, 
using formative assessment, incorporating literacy into content classrooms, and increasing the 
rigor of their writing assignments. At the same time, although teachers felt that LDC had 
benefited their students’ writing and college readiness, less than half reported that their students 
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had a great deal of success on their LDC module reading and writing mini-tasks or on the final 
writing task. These results suggest that participating teachers may have needed additional help 


with the design and implementation of the modules and with their LDC instruction. 
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Chapter 4: Student Learning Results 


The study used multiple measures of student learning both to examine LDC effects and to 
explore relationships between LDC implementation variables and student outcomes. Below we 
first provide descriptive results for both the CRESST ILAs and state assessment measures for the 
LDC sample only. The results of the quasi-experimental analysis of LDC effects on learning in 


Kentucky then follow. 
Descriptive Results 


Evolution and Reconstruction ILAs. As noted earlier, subject-specific CRESST-designed 
Integrated Learning Assessments (ILAs) were administered in Kentucky and Pennsylvania study 
classrooms. The social studies ILA focused on Reconstruction following the Civil War, while the 
science ILA assessed scientific thinking in the context of evolution. Each ILA included both a set 
of multiple choice and short answer items that addressed reading comprehension and analysis, 
and a final essay that was scored on five dimensions, each using a 1-4 scale. Writing results 


were summarized as a total score across the five dimensions and by dimension. 


Table 14 and Table 15 display descriptive statistics for the total performance of students on 
the ILAs in Kentucky and Pennsylvania respectively for the reading and writing portions of the 
assessments. These results show considerable variation in performance across students in both 
States and assessments. However, mean reading and writing scores on both assessments were 
quite low. In reading, for both states, students on average earned roughly half of the total 
possible score points on both the Evolution and Reconstruction assessments. On the writing 
component, the mean scores ranged between 36% and 43% of the total possible, depending on 


the state and assessment. 


Table 14 


Descriptive Results of ILAs Administered in Kentucky 


nof Total possible Mean 
Assessment Component students score score SD Minimum Maximum 
Evolution Reading 166 15 7.59 3.13 0 15 
Evolution Writing 132 20 7.63 2.61 5 17 
Reconstruction Reading 252 18 9.90 2.90 1 16 
Reconstruction Writing 253 20 8.56 2.89 5 19 
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Table 15 


Descriptive Results of ILAs Administered in Pennsylvania 


nof Total possible Mean 
Assessment Component students score score SD Minimum Maximum 
Evolution Reading 222 15 7.49 3.16 0 14 
Evolution Writing 203 20 7.29 2.37 5 16 
Reconstruction Reading 197 18 9.98 3.53 1 17 
Reconstruction Writing 205 20 7.87 2.69 5 17 


As can be seen in Table 16 and Table 17, which combine data across states, student 
performance on the five writing dimensions was generally similar. On both ILA assessments, 
average dimension scores fell between Levels 1 and 2. Grammar and conventions appears to 
have been a relative strength. Mean scores on the Reconstruction writing task appear slightly 
higher than scores on the Evolution writing task. However, the two tasks are not directly 
comparable nor are study samples representative, so no inferences can be drawn about students’ 


relative success in the two subject areas. 


Table 16 


Descriptive Statistics for ILA Evolution Writing Task Score Dimensions 


Dimension n Mean SD Minimum Maximum 
Content understanding 335 1.30 0.58 1 4 
Rhetorical structure/quality 335 1.44 0.59 1 3 
Organization 335 1.54 0.75 1 4 
Reference/support with text 335 1.41 0.58 1 4 
Grammar and conventions 335 1.73 0.70 1 4 
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Table 17 


Descriptive Statistics for ILA Reconstruction Writing Task Score Dimensions 


Dimension n Mean SD Minimum Maximum 
Content understanding 458 1.57 0.70 1 4 
Rhetorical structure/quality 458 1.68 0.67 1 4 
Organization 458 1.68 0.76 1 4 
Reference/support with text 458 1.48 0.65 1 4 
Grammar and conventions 458 1.84 0.62 1 4 


Kentucky Performance Rating for Educational Progress (K-PREP) results. As 
described earlier, the Kentucky study drew on eighth-grade students’ end-of-year performance 
on three K-PREP assessments: reading, writing, and social studies. Table 18 displays descriptive 
statistics for these performance data at the end of the study year, 2012-2013, and the prior year. 
The 2012-2013 results in reading, writing, and social studies suggest that study LDC students 
scored just above the state mean in reading and social studies and just below it in writing, but 
differences are negligible. In the year prior, sampled students’ performance was slightly above 
the state mean in reading and slightly below it in social studies, but again the differences are 
negligible (3 scale score points). (See Pearson, 2012, 2013, for data on statewide Grade 8 


results.) 


Table 18 


LDC Students’ K-PREP Performance for Study and Prior Years 


Variable n Mean SD Minimum Maximum 

K-PREP reading 

Study year, 2012-2013 2529 213.27 15.23 157 278 

Prior year, 2011-2012 2529 211.34 15.84 162 274 
K-PREP writing 

Study year, 2012-2013 only 2529 10.02 2.59 0 16 
K-PREP social studies 

Study year, 2012-2013 2529 215.94 17.01 109 300 

Prior year, 2011-2012 2529 218.36 15.74 114 300 


Note. Scores are for LDC students as eighth-graders in 2012-2013, and seventh-graders in 2011-2012. 


Before moving to our analysis of the extent to which LDC and features of it influenced 


student learning, we report on the correlations between the five study measures addressing 
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student learning outcomes: ILA writing, ILA reading, K-PREP reading, K-PREP writing, and 
K-PREP social studies, all of which were administered in spring of the study year. As one would 
expect from measures addressing different constructs, correlations shown in Table 19 and Table 
20 are moderate. The relatively highest correlations, .6 and above, are between the different 
K-PREP measures. Moderate correlations were also found between reading-oriented K-PREP 
measures (K-PREP reading and K-PREP social studies) and ILA reading. K-PREP writing and 
ILA writing show a relatively low correlation. While this may seem surprising at first glance, 
there are two explanations. First, both measures are based on a small sample of tasks—one for 
the ILA and two for the K-PREP direct writing assessment. Substantial research indicates the 
very limited generalizability of individual student scores on such tests (Cronbach, Linn, Brennan, 
& Haertel, 1995; Gao, Shavelson, & Baxter, 1994; Resnick, Resnick, & DeStefano, 1993). That 
is, students’ writing performance is likely to vary with different topics and types of writing tasks, 
so it takes many tasks to get a reliable estimate of student writing. As a result, both K-PREP and 
ILA writing scores contain substantial error, which depresses correlations between the two. 
Second, the two assessments are conceptually different. Only one of the two K-PREP writing 
tasks is passage based, and it involves only one passage to stimulate writing, while the ILA 


involves the synthesis of multiple texts with background knowledge. 


Table 19 


Correlation Between Evolution ILA and State Assessments 


EvolutionILA EvolutionILA 2013K-PREP 2013K-PREP 2013 K-PREP 
writing score reading score reading writing social studies 


Evolution ILA writing score — 


Evolution ILA reading score 0.61 (121) — 


2013 K-PREP reading 0.61 (121) 0.56 (151) _ 
2013 K-PREP writing 0.58 (121) 0.51 (151) 0.74 (152) = 
2013 K-PREP social studies 0.58 (121) 0.55 (151) 0.80 (152) 0.68 (152) =4 


Note. n presented in parentheses. 
All correlations significantly different from zero (p < .0001). 
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Table 20 


Correlation Between Reconstruction ILA and State Assessments 


Reconstruction Reconstruction 
ILA writing ILA reading 2013K-PREP 2013 K-PREP 2013 K-PREP 
Assessment score score reading writing social studies 


Reconstruction ILA writing — 
score 


Reconstruction ILA reading 0.45 (233) — 

score 

2013 K-PREP reading 0.54 (242) 0.60 (239) — 

2013 K-PREP writing 0.46 (242) 0.37 (239) 0.60 (252) — 

2013 K-PREP social studies 0.49 (242) 0.62 (239) 0.68 (252) 0.46 (252) — 


Note. n presented in parentheses. 
All correlations significantly different from zero (p < .0001). 


Quasi-Experimental Analysis of LDC Effects in Kentucky 


This section presents the results of our quasi-experimental design analysis of the impact of 
LDC in social studies and science classes on student learning in Kentucky. We begin by 
describing the treated teacher and student samples for the analysis. We then summarize the 
matching process we used to select similar comparison students and to control for the prior 
effectiveness of teachers and schools. Next we outline the structure and design of the two 
hierarchical linear models (HLMs) we employed to estimate the impact of LDC. Finally we 
present the results of LDC’s impact on three outcome measures—K-PREP reading, writing, and 


social studies—using the two modeling approaches. 


Teacher and student sample. As described earlier, our LDC teacher sample includes all 
eighth-grade social studies and science teachers in the five target Kentucky school districts who 
began teaching LDC in either 2010-2011 or 2011-2012 and continued implementing LDC in 
2012-2013. This group included 37 teachers, of whom seven are Phase 1 (began LDC 
participation in 2010-2011) and 30 are Phase 2 (began participation in 2011-2012). As we 
explain further below, our analyses, where possible, control for the prior effectiveness of 
teachers by calculating the “value added” to their students using assessment scores from 2008- 
2009 and 2009-2010 (prior to the start of the LDC initiative). These data were available for five 
of the seven Phase 1 teachers, and 17 of the 30 Phase 2 teachers. Table 21 summarizes the phase 


participation and availability of prior effectiveness data for our group of treatment teachers. 
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Table 21 


Treatment Teacher Sample by Phase Participation and Availability of Prior 
Effectiveness Data 


Data available for prior effectiveness 


Yes No 
Phase 1 (began participation in 2010-2011) 5 2 
Phase 2 (began participation in 2011-2012) 17 13 


The eligible student sample for the analysis includes all students (a) who were enrolled in 
an eighth-grade social studies or science class taught by one of the 37 teachers, and (b) for whom 
prior achievement scores were available. This sample includes 2,529 students and is described in 
Chapter 2 of this report. As noted in Chapter 2, these students are quite similar to all students 
statewide on both demographic and student achievement variables. The treatment sample does 
have a higher proportion of White students, lower proportion of Black students, and slightly 
lower proportion of students eligible for free or reduced price lunch than the population of 
students statewide. 


Selection of comparison students. Treatment students and teachers were not randomly 
selected to participate in the LDC initiative. To estimate the impact of LDC it is therefore 
necessary to control for the effects of student, teacher, and school characteristics. One way to 
control for these characteristics is to use matching techniques to identify a group of comparison 
students who are demographically and academically similar to the intervention students. Our 
matching is conducted at the student level, and accounts not only for student demographics and 


prior achievement, but also the prior effectiveness of teachers and schools as well. 


We employ a matching technique known as Coarsened Exact Matching (CEM) to identify 
comparison students. Coarsened Exact Matching is a flexible matching approach with many 
favorable properties, and allows the researcher to specify the precise conditions under which a 
comparison student may be matched with an intervention student. For categorical variables such 
as race/ethnicity or free/reduced price lunch status, this often entails exact matching, while for 
continuous measures, such as prior outcomes and prior teacher effectiveness scores, cut-points 
for matching can be specified. With this approach we can set precise cut-points on the most 
important prior indicators such as prior academic achievement to ensure that where possible 


every treatment student is matched with a suitable comparison. 


This process was applied for each of the three outcome measures, resulting in three 


matched datasets. Creating separate matched datasets for each outcome maximized the sample 
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size for each outcome analysis as patterns of missing data varied across outcome measures. 
Table 22 summarizes the variables used for the matching. Please note that although we include 
indicators for students, teachers, and schools, all matching is at the student level. Student 
characteristics in the model include a number of demographic variables (race/ethnicity 
categories, gender, free and reduced price lunch eligibility, etc.) as well as prior achievement on 
two state assessments (reading and science). In addition to controlling for these student 
characteristics, our matching methodology also selected comparison students whose teachers had 
similar prior effectiveness. Prior effectiveness was produced by calculating a teacher’s value 
added on student learning in 2009-2010. The assessments used for this variable depended on the 
outcome measure we were testing; the matching model used to test the impact of LDC on writing 
used writing scores for 2008-2009 and 2009-2010 to calculate prior teacher effectiveness, and 
likewise for reading and social studies. Students under teachers without prior effectiveness data 
were matched to comparison students under teachers with missing data as well (most of these 
teachers were likely new to the profession). Finally, a school prior effectiveness variable was 
calculated using prior seventh-grade science, math, and reading assessment data. Seventh-grade 
data were used to ensure that the school effectiveness variable was independent of the teacher 


effectiveness variable in the matching model. 


Table 22 


Summary of Matching Variables 


Indicator type Variable 
Student Gender 

Student White 

Student Hispanic 

Student Black 

Student Asian 

Student Special education 

Student Free/reduced price lunch eligible 
Student Title I 

Student English language learner 
Student Prior achievement in reading 
Student Prior achievement in science 
Teacher Availability of teacher prior effectiveness data 
Teacher Teacher prior effectiveness 
School School prior effectiveness 
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The CEM process was successful in finding similar matches for a large majority of the 
eligible 2,529 LDC students. Ninety-one percent of the treatment students were retained in the 
sample for the writing outcome analysis after matching, as were 88% of treatment students for 
the reading analysis and 90% for the social studies analysis. See Table 23 for a summary of the 
number of treatment and control students before and after matching. The matching models were 
effective in achieving close balance with regard to prior student scores and demographics, as 
well as for the teacher and school effectiveness indicators (see Tables E1 through E3 in 
Appendix E for prior achievement and demographic characteristics of eligible and matched 
treatment and comparison samples for each outcome; for ease of interpretation, we display only 
student characteristic variables in these Appendix tables and leave out teacher and school 
effectiveness variables). 


Table 23 


Summary of Treatment and Comparison Samples by Outcome 


Sample Treatment Comparison 
Eligible for matching 2529 43333 
Matched sample for writing 2300 12208 
Matched sample for reading 2232 13174 
Matched sample for social studies 2284 18265 


Modeling approach. For each of the three outcome measures, two separate two-level 
hierarchical linear models (HLMs) are employed. Each HLM attempts to model students’ dosage 
under treated and non-treated teachers in eighth-grade science and social studies courses. In each 
model, where possible, measures of teacher effectiveness on the outcome measure of interest 
prior to the LDC intervention are estimated and used as value-added controls. Student 
demographic and prior achievement variables, and teacher and school prior effectiveness are also 
included in the models. Our estimates therefore control for observables in two ways, at the 
matching and modeling stages. The models also examined potential interactions between the 
LDC treatment and prior school and teacher effectiveness as well as student characteristics. 
These interaction variables were intended to test whether LDC had differential effects on student 
learning depending on the school, teacher, and/or individual student’s standing on the given 
variable. These interaction analyses should be considered highly exploratory and results treated 
as tentative. 
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Table 24 summarizes how observations are defined at each level in the two HLMs. In 
Model 1, Level 1 observations are student/course combinations. As a result each student can be 
and likely is represented multiple times at Level 1. A weight is applied to Level 1 observations 
so that each student’s science course(s) cumulatively sum to 0.5, and each student’s social 
studies courses cumulatively sum to 0.5. Thus a student who took some combination of science 
and social studies courses will receive a cumulative weight of 1. One individual teacher is 
associated with each student/course observation and thus each individual teacher is an 
independent observation at Level 2. 


In Model 2, each observation at Level 1 represents one student. Level 2 observations 
represent the combination of a social studies and science teacher. To simplify the design, 
students with more than one science or social studies teacher were randomly assigned one of 
those multiple teachers. Therefore, each Level 1 observation is associated with one Level 2 
observation. This should not present a significant problem as a substantial majority of students in 
2012-2013 were associated with only one science and one social studies teacher. Prior teacher 
and school effectiveness indicators were aggregated as cumulative sums for the teacher 
combination at Level 2. 


Table 24 


Observations by Level for Two Hierarchical Linear Models 


Level Model 1 Model 2 
Level 1 Student/course combination Student 
Level 2 Teacher Social studies and science teacher combination 


Each of the two models has advantages. In Model 1, it is not necessary to remove any 
teacher observations. However, the repetition of students at Level 1 is somewhat nonstandard 
and therefore the standard errors may be underestimated. On the other hand, the structure of 
Model 2 requires that a small amount of information on teacher impact be eliminated, but the 
structure of the model is more standard, and therefore we have a higher level of confidence 
regarding the standard errors. Overall we favor Model 2 given greater confidence regarding 
standard errors, and we therefore choose to display those results in the next section. Detailed 
results from both models, which show a high level of consistency across model specifications, 
are displayed in Appendix Tables E4 through E9. 


Further, note that prior teacher effectiveness was a variable of interest but was missing for 


some teachers because they were relatively new to the system or were not teaching at the same 
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grade at the prior time point. Our Coarsened Exact Matching process matched treatment students 
under LDC teachers whose data were missing with comparison students whose teachers also had 
missing scores; the missing teacher effectiveness scores were set at zero. Because we have more 
confidence in the match for those teachers who were not missing information from the period 
prior to intervention, we created and tested the effect of missing and the interaction between 
treatment and teachers who were missing prior data; we also tested the joint significance of the 
main LDC effect and its interaction with missing since we lose some power to find an overall 
effect by testing these effects separately. These analyses showed no significant main or 
interaction effects for missing and therefore these variables were excluded from subsequent 


analyses. 


HLM results of the impact of LDC on student learning. HLM results for Model 2 for 
each of the three primary outcomes are displayed in Table 25, Table 26, and Table 27. The 
models shown here include a number of interactions between treatment status and student 
characteristic variables, as well as the interaction between treatment status and the prior 
effectiveness of the teacher. Results for Model 1 and for models not including the interaction 
terms are presented in Tables E4 through E9 in Appendix E. It should be noted that in Model 1 at 
Level 2 each teacher is coded as 1 if s/he was in the LDC intervention and zero if not. In Model 
2, each teacher combination observation at Level 2 would receive a value of zero if neither 
teacher were treated, 1 if one of the two were treated, and 2 if both teachers were treated. Thus 
the treatment effect coefficients for each model represent the effect of one treated teacher. While 
the value-added models controlled for all of the student, teacher, and school indicators 
previously discussed, we limit our presentation in the body of the report to the intervention 
effects of interest. Table 25 shows HLM results for the K-PREP reading scores. The data 
indicate that LDC had a small statistically significant, positive effect on students’ reading 
performance. LDC students scored higher in reading than did their carefully matched comparison 


group, demonstrating that LDC had a measurable effect on students’ literacy learning. 


To provide a benchmark for interpreting this effect, we used a relatively new methodology 
to convert the effect size into a gross indicator of the number of months of learning it represents 
(see Hill, Bloom, Black, & Lipsey, 2007). Following this approach, we used available data to 
estimate the growth in K-PREP reading scores from eighth to ninth grade. We then determined 
the proportion of typical growth represented by the observed LDC effect size—that is, the LDC 
effect size divided by the effect size expected from Grade 7 to Grade 8. We then used this 
proportion to calculate the number of months, relative to a nine-month academic year, the 
additional growth associated with LDC. Relative to typical growth in reading from eighth to 
ninth grade, the calculation found that the effect size for LDC represents 2.2 months of 


42 


schooling. Given that a typical Kentucky teacher spent four to eight weeks teaching LDC, it 
appears that LDC was effective in achieving literacy gains in a shorter period of time than 


regular instruction. 


Table 25 


2012-2013 LDC Student Effect Estimates on K-PREP Reading, Including 
Interactions With Prior Teacher Effectiveness and Student Characteristics 


Level 2 variables Model coefficient (SE) 

LDC treatment 0.058 (0.023)* 
LDC treatment by teacher effectiveness -0.181 (0.202) 
Level 1 treatment by student characteristic interactions 

Gender -0.004 (0.017) 

Special education -0.110 (0.034)* 

Free/reduced price lunch eligible 0.053 (0.017)* 

Prior achievement 0.034 (0.011)* 


Note. Fixed effects for demographic predictors and for prior school and teacher 
effectiveness not shown. 
*p = .05. 


The data also show interactions between LDC effects and student characteristics. Both 
students’ prior achievement, based on their prior year K-PREP scores, and_ students’ 
socioeconomic status (SES), as revealed by their free or reduced price lunch status, show 
positive interactions with the treatment. That is, LDC students who were relatively higher 
achieving prior to their LDC experience showed relatively greater benefit than did those who 
started relatively lower achieving, although the observed effect is very small. Interestingly, LDC 
students eligible for free or reduced price lunch also appeared to have benefited more from LDC, 
after controlling for other variables. Although, again, the observed effect was very small, we 
speculate that LDC students with lower SES status perhaps had access to special resources (e.g., 
Title I programs, specialist teachers) that provided essential support. We did not find evidence of 
differential effects of LDC by gender. Controlling for other factors, special education students 
appeared to do less well under LDC; however the share of students falling into this category was 


small. 


The results for K-PREP social studies are shown in Table 26. The coefficient for the main 
effect for LDC is small and not statistically significant, indicating that LDC’s addition of literacy 
to course requirements did not diminish students’ content performance. Table 26 also reveals a 


significant interaction between prior teacher effectiveness and LDC. LDC students taught by 
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teachers who were relatively less effective prior to LDC benefited more than did students of 
relatively more effective teachers. However, this interaction is difficult to interpret and should be 
treated cautiously given that all teachers’, including science teachers’, prior effectiveness scores 
were based on their students’ eighth-grade social studies performance for the study’s baseline 


year (because Kentucky does not assess science in eighth grade). 


Students’ prior year performance on the K-PREP and their free or reduced price lunch 
status show the same, small positive interaction with LDC treatment status as in the reading 
outcome model. LDC students who started the year performing at a relatively higher level 
experienced more benefit from LDC in their social studies performance, as did students who 
were from a relatively lower SES, as evidenced by their free or reduced price lunch status. We 


did not find differential treatment effects of LDC by gender or special education status. 


Table 26 


2012-2013 LDC Student Effect Estimates on K-PREP Social Studies, Including 
Interactions With Prior Teacher Effectiveness and Student Characteristics 


Level 2 variable Model coefficient (SE) 

LDC treatment -0.026 (0.023) 
LDC treatment by teacher effectiveness -0.288 (0.082)* 
LDC treatment by student characteristics interactions 

Gender 0.013 (0.016) 

Special education -0.007 (0.037) 

Free/reduced price lunch eligible 0.039 (0.019)* 

Prior achievement 0.050 (0.017)* 


Note. Fixed effects for demographic predictors and for prior school and teacher 
effectiveness not shown. 
*p = .05. 


K-PREP writing results, as shown in Table 27, show neither main nor interaction effects 
for LDC. There is no evidence of any impact of the LDC intervention on this particular writing 
assessment. 
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Table 27 


2012-2013 LDC Student Effect Estimates on K-PREP Writing, Including 
Interactions With Prior Teacher Effectiveness and Student Characteristics 


Level 2 variable Model coefficient (SE) 

LDC treatment 0.030 (0.042) 
LDC treatment by teacher effectiveness 0.004 (0.120) 
LDC treatment by student characteristics interactions 

Gender -0.032 (0.031) 

Special education 0.031 (0.047) 

Free/reduced price lunch eligible -0.002 (0.027) 

Prior achievement 0.016 (0.016) 


Note. Fixed effects for demographic predictors and for prior school and teacher 
effectiveness not shown. 
*p = .05. 


Summary of Student Learning Results 


In summary, the Kentucky HLM results suggest a small positive LDC treatment effect on 
K-PREP reading scores, an effect size which translates into approximately 2.2 months of 
instruction based on available methodology. Neither K-PREP social studies nor K-PREP writing 
scores provide any evidence of a treatment effect in either direction. In both reading and social 
studies, the analyses found positive interaction effects for students’ prior achievement and 
free/reduced price lunch status and a negative interaction with students’ special education status. 
These findings suggest that initially higher performing students received relatively more benefit 
from LDC than did initially lower performing students and that lower SES status, as indicated by 
free and reduced price lunch status, was associated with higher scores. Special education 
students appeared to derive less benefit from LDC. In addition, the social studies analysis 
revealed a negative interaction with prior teacher effectiveness, indicating that students whose 
teachers were initially relatively more effective showed less benefit than their peers with teachers 


who were initially less effective. 


The Pennsylvania data show no evidence of any effect of LDC. However, the limitations of 


the available data render these analyses inconclusive. 
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Chapter 5: Implementation Variables Related to LDC Success 


In this chapter, we report on analyses of the relationship between variables derived from 
our implementation measures and student outcomes. Drawing on data from the three 
implementation measures—teacher log, teacher survey, and LDC module analysis—we explored 
a variety of composite implementation variables and examined their relationship to student 
learning outcomes. Because of data limitations in Pennsylvania, the analysis focuses on 
Kentucky teachers. When more standard regression analyses failed to produce stable patterns of 
results, we investigated the extent to which a variety of implementation variables differentiated 
LDC teachers at relatively high, middle, and low levels of effect on student learning, based on 
the estimated value added of teachers during the study year. We then compared the mean scores 
on each variable using an ANOVA difference in means test. These processes are further 
described in the following section. Because these analyses are exploratory, particularly given the 


small sample sizes, results should be interpreted with caution. 
Identification of Implementation Variables 


Both substantive theory and psychometric analysis guided the development of composite 
variables. Our identification of priority variables centered on evidence-based teacher practices 
that were likely to influence student learning (e.g., Heritage, 2010; Herman, Osmundson, Dai, 
Ringstaff, & Timms, 2011; Hinchman & Sheridan-Thomas, 2008) and on variables that 
influence the implementation of new practices and programs—for example, teacher beliefs, 
sense of efficacy, leadership support, collaboration, and professional development (see for 
example, Fullan, Hargreaves, & Lieberman, 2010; O’Day, Bitter, & Gomez, 2011; Supovitz & 
Weinbaum, 2008). Through cycles of hypothesis generation and a variety of exploratory factor 
analyses (EFA) and cluster analyses, we identified 19 variables for additional study. 


These variables, their sources, and operational definitions are shown in Table 28. 
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Table 28 


Teacher-Level Implementation Variables Used in Within-Treatment Analyses 


Instrument 
source Variable Description 

LDC module Overall module An overall module quality score was created as the mean score across 

measure quality the nine dimensions for each module, averaged for the two modules 

submitted by each teacher. 

Teacher log Range and intensity of | Sum of reading skills reported for each teacher log in which reading 
reading instruction was addressed. A mean total score for each teacher was then 

computed as the average across all relevant logs. Scores for each 
reading skill indicated the emphasis it was given that day: focus of 
student work = 2; touched on briefly = 1; not today = 0. 

Teacher log Attention to close Mean sum of reading skills reported representing high-level analysis 

reading of text of text on logs for which reading was addressed. Scores for each close 
reading item indicated the emphasis it was given that day: focus of 
student work = 2; touched on briefly = 1; not today = 0. 

Teacher log Attention to basic Mean sum of basic reading skills items reported on logs for which 

reading skills reading was addressed. Scores for each basic reading skill item 
indicated the emphasis it was given that day: focus of student work = 
2; touched on briefly = 1; not today = 0. 

Teacher log Attention to writing Mean sum of all writing skills reported on logs for which writing was 

skills addressed. Coding: focus of student work = 2; touched on briefly = 1; 
not today = 0. 

Teacher log Range and intensity of Mean sum of all formative assessment practices reported in logs 
formative assessment addressing reading and/or writing. Scores for each formative 
of student learning assessment practice indicated the extent to which it was used: to a 

great extent = 2; to some extent = 1; not at all = 0. 

Teacher log Range and intensity of Mean sum of all practices for providing feedback to students based on 

feedback to students student work in reading and writing. Scores for each feedback practice 
indicated the extent to which it was used: to a great extent = 2; to 
some extent = 1; not at all = 0. 

Teacher log Range and intensity of | Continuous variable measuring the extent to which teachers reported 
teacher literacy attention to: reading skills, writing skills, formative assessment 
practices (reading practice, and providing feedback. Each of these four domains was 
skills, writing skills, weighted equally to create the variable. 
formative assessment) 

Teacher log Teacher log cluster Cluster binary variable distinguishing teachers who reported 


Teacher survey 


variable: High 
quantity literacy 
practice 


Factor 1: Attention to 
close reading of text 


conducting a greater quantity of practices during LDC module 
instruction from teachers who reported a smaller quantity of practices. 
Variable was created by first performing cluster analysis on individual 
items in each domain (reading skills, writing skills, formative 
assessment), and then performing a second cluster analysis using the 
identified cluster variables. 


Factor 1 derived from exploratory factor analysis including teacher 
survey items on attention to reading skills, writing skills, and use of 
formative assessment. Factor 1 reflected reading items related to close 
reading of text. Variable confirmed and tested for reliability using 
confirmatory factor analysis. 
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Instrument 
source 


Variable 


Description 


Teacher survey 


Teacher survey 


Teacher survey 


Teacher survey 


Teacher survey 


Teacher survey 


Teacher survey 


Teacher Survey 


Teacher survey 


Factor 2: Attention to 
paragraph 
writing/structure 


Factor 3: Teacher-led 
formative assessment 
practice 


Factor 4: Peer- 
oriented formative 
assessment practice 


Total modules taught 
in 2011-2012 and 
2012-2013 school 
years 


Support for teaching 
literacy in content 
area classrooms 


Teachers’ perceived 
capacity to teach LDC 


District and school 
support for LDC 


Utility of teacher 
collaboration 


Professional 
development dosage 


Factor 2 derived from exploratory factor analysis including teacher 
survey items on attention to reading skills, writing skills, and use of 
formative assessment. Variable reflected writing items related to 
paragraph construction and structure of writing. Confirmed and tested 
for reliability using confirmatory factor analysis. 


Factor 3 derived from exploratory factor analysis including teacher 
survey items on attention to reading skills, writing skills, and use of 
formative assessment. Variable reflected teacher-oriented formative 
assessment practices. Confirmed and tested for reliability using 
confirmatory factor analysis. 


Factor 4 derived from exploratory factor analysis including teacher 
survey items on attention to reading skills, writing skills, and use of 
formative assessment. Variable reflected student-to-student formative 
assessment practices. Confirmed and tested for reliability using 
confirmatory factor analysis. 


Sum of responses to teacher survey Questions 16 and 17. 


Mean across three items addressing content teachers’ time and 
responsibility for teaching literacy. Coding: disagree = 0; disagree 
somewhat = 1; agree somewhat = 2; agree = 3. 


Mean response to questions about teacher efficacy (Question 26) and 
barriers with regard to LDC (Questions 39). Coding: disagree = 0; 
disagree somewhat = 1; agree somewhat = 2; agree = 3, with Items 
39c—g reverse coded. 


Mean response to items about various ways that district and school 
leadership show support for LDC (Question 43). Coding: disagree = 
0; disagree somewhat = 1; agree somewhat = 2; agree = 3. 


Mean response to items asking about extent and helpfulness of teacher 
collaboration in implementing LDC (Question 49). Coding: disagree 
= 0; disagree somewhat = 1; agree somewhat = 2; agree = 3. 


The number of formal scheduled LDC PD sessions in 2012—2013 
(Question 55). 


As the table shows, the implementation analyses included a variable representing overall 


module quality, which was calculated by taking the mean of the nine quality dimension scores 


and then averaging across modules to the teacher level. As noted in the appended report on the 


CRESST Assignment Measure, factor analyses indicated that all the module dimensions load on 


a single factor, supporting the claim that the CRESST Assignment Measure effectively measures 


a single coherent trait. Analyses in this chapter thus are limited to the mean dimension score and 


do not test differences on individual dimensions. 
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A number of teacher-level implementation variables were created from teacher log 
responses. The variables are summary measures of teacher responses in four key domains of the 
log, which also represent component emphases for LDC: teachers’ daily focus on reading skills, 
teachers’ daily focus on writing skills, teachers’ daily use of strategies to assess student learning, 
and teachers’ daily use of strategies to provide feedback to students. The latter two domains 
together constitute our measure of formative assessment practice. Mean sum variables were 
created for each domain, based on both the number of skills or strategies the teacher reported 
when reading and/or writing was addressed and the depth of attention reportedly given to the 
skill (e.g., on the writing variable, a writing skill would be coded as 2 if the teacher reported it 
was a primary focus on the day of the log, and a 1 if the teacher reported that it was only touched 
on briefly). For the reading skills domain, we also separated mean sum variables for two 
subgroups of items: those emphasizing close reading of text and those addressing more basic 
reading skills. The decision to analyze this domain at a finer level of detail was based on both 
our theoretical assumptions regarding the relative importance of skill development in these two 
areas and exploratory analysis of the log and survey data that provided evidence of the 
dichotomy. 


Finally, we included two variables that attempt to capture variety in the teachers’ reported 
attention to all four domains. One variable is the mean sum of activity reported across all four 
domains described above, with equal weighting given to each. The second variable is a binary 
indicator derived from cluster analysis, a statistical methodology that creates a specified number 
of teacher groups based on the association of teacher responses to a series of items. We 
conducted separate cluster analyses for each of the above four domains. In each case, the derived 
clusters separated teachers into two groups: a high group that was high in reported practice in 
each domain and a low group, which represented those who reported a smaller sum of practice. 
We then conducted a second cluster analysis using the derived cluster variables. The final cluster 
variable is a binary variable (i.e., coded 1/0) that distinguishes two groups based on the 
individual cluster scores. The first group reflected teachers who were high on all the individual 
clusters and the second, teachers who were low on all the individual clusters. The two clusters 
thus represent teachers who more extensively implemented targeted practices in each domain 


(coded 1) versus those whose implementation was relatively less extensive. 


The teacher survey variables include four factors derived from an exploratory factor 


analysis.* This factor analysis included all survey items designed to parallel the log reports in 


*Note that the exploratory factor analysis was performed on a larger sample of teachers that included teachers in 
Pennsylvania and from our parallel study of sixth-grade reading in Florida. Reliability analyses focused just on the 
Kentucky teachers and suggested that the constructs held for the smaller sample of teachers. 
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four key domains: reading skills, writing skills, assessing student learning, and feedback. 
Exploratory factor analysis clustered items in four theoretically distinct factors which we 
characterized as attention to close reading of text, paragraph writing/structure, teacher-led 
formative assessment practice, and peer-oriented formative assessment practice. We then tested 
the reliability of the identified factors. As can be seen in Table 29, reliability was high for each 
of the factors, including Factor 4, which had relatively few items. Other survey variables include 
the total number of modules taught in 2011-2012 and 2012-2013 (a measure of teacher LDC 
experience), a measure of teachers’ commitment to teaching literacy in content area classes, a 
measure designed to capture teachers’ perceived capacity to teach LDC, perceived district and 
school support for LDC, the perceived utility of teacher collaboration around LDC, and a 


variable measuring the amount of professional development received. 


Table 29 


Reliability of Teacher Survey Factors 


Factor Description Number of items Cronbach’s alpha 

1 Close reading 10 89 

2 Paragraph writing/structure 4 84 

3 Teacher-led formative assessment strategies 11 83 

4 Peer-oriented formative assessment strategies 3 74 
Methodology 


The teacher sample for this analysis included the 17 Kentucky teachers with complete data 
for each of the three measures (assignment, log, and survey). The analysis took part in two 
stages. First HLM models were used to calculate the value added for each teacher and break the 
teachers into three groups based on their effectiveness. Second, ANOVA difference in means 
tests were used to see if there was a statistical difference between the groups on each of the 
chosen implementation variables. The student sample included 352 students from the 17 
teachers’ classes for whom we have valid 2012 K-PREP pre-scores, 2013 K-PREP outcome 
scores, and reading and writing scores on a CRESST ILA. The ILA portion of the outcome 
measure was based on either the Reconstruction or Evolution topic, depending on whether a 


given student was participating in an LDC classroom in history/social studies or science. 


HLM was used to classify teachers into three levels of relative effectiveness, teachers 
whose students achieved relatively high, medium, and low levels of performance during the 


study year. The analysis controlled for students’ prior year, 2012 performance on K-PREP 
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reading and science assessments and used a composite outcome measure to determine teachers’ 
relative value added. Scores on the four available 2013 measures—2013 K-PREP reading, 2013 
K-PREP writing, ILA reading, and ILA writing—were standardized and then averaged to create 
a more robust overall measure. As noted in Chapter 2, correlations between K-PREP and ILA 
scores are moderate, which is not surprising given differences in the design and intended 
learning targets of the two measures. By averaging the scores across the two, we sought to 
capture a fuller and more reliable picture of student learning than any of our measures would 
individually—for example, K-PREP which is not as well aligned to LDC, and the ILAs which 
are better aligned but show lower reliability than the K-PREP. 
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Table 30 


Mean Scores on Implementation Variables for Teachers With Low, Medium, and High Value Added and ANOVA Test of Difference in Means 


Low value added Medium value added High value added 
(n=5) (n=7) (n=5) 
Instruments Variable n Mean SD Mean SD Mean SD Test statistic |p value 
Assignment Overall module quality 5 3.27 0.53 3.05 0.80 3.63 0.89 0.84 0.45 
measure 
Teacher log Range and intensity of reading 4 6.73 2.56 11.93 7.08 10.78 3.30 1.26 0.32 
instruction 

Teacher log Attention to close reading of 4 1.63 1.49 3.95 2.26 3.10 1.55 1.91 0.19 
text 

Teacher log Attention to basic reading skills 4 5.10 1.71 7.98 4.91 7.68 1.79 0.90 0.43 

Teacher log Attention to writing skills 4 7.06 1.20 12.27 6.93 12.37 5.91 1.24 0.32 

Teacher log Range and intensity of 4 7.85 27 10.85 5.56 8.86 8.24 0.35 0.71 
formative assessment of 
student learning 

Teacher log Range and intensity of 4 7.96 5.69 15.05 7.04 12.25 12.94 0.78 0.48 
feedback to students 

Teacher log Range and intensity of teacher 4 5.42 1.38 9.29 3.96 8.43 4.27 1.47 0.27 
literacy practices (reading 
skills, writing skills, formative 
assessment) 

Teacher log Teacher log cluster variable: 4 0.25 0.50 0.86 0.38 0.60 0.55 0.04 0.15 
high quantity literacy practice 

Teacher survey __- Factor 1: Attention to close 3 -0.57 0.23 0.14 0.93 -0.11 0.54 0.97 0.41 
reading of text 

Teacher survey _- Factor 2: Attention to 3 -0.45 0.50 0.15 0.88 0.02 0.38 0.79 0.48 
paragraph writing/structure 

Teacher survey _ Factor 3: Teacher-led formative 3 -0.54 1.01 0.50 0.75 0.01 0.63 2.04 0.17 


assessment practice 
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Low value added Medium value added High value added 
(n=5) (n=7) (n=5) 
Instruments Variable Mean SD n Mean SD Mean SD Test statistic  p value 

Teacher survey __ Factor 4: Peer-oriented -0.62 0.83 7 -0.10 1.10 -0.49 0.55 0.45 0.65 
formative assessment practice 

Teacher survey Total modules taught in 2011— 3.33 0.58 vi 3.86 1.07 3.60 0.55 0.42 0.67 
2012 and 2012-2013 school 
years 

Teacher survey Support for teaching literacy in 2.22 0.19 7 2.29 0.30 2.07 0.15 1.21 0.33 
content area classrooms 

Teacher survey ‘Teachers’ perceived capacity to 1.03 0.56 7 1.53 0.40 2.00 0.67 3.18 0.08 
teach LDC 

Teacher survey _ District and school support for 1.58 0.38 6 2.13 0.48 2.20 0.54 1.67 0.23 
LDC 

Teacher survey _ Utility of teacher collaboration 1.88 1.07 7 2.55 0.56 2.58 0.95 0.88 0.44 

Teacher survey Professional development 1.67 1.53 7 2.29 1.11 3.80 1.30 3.31 0.07 


dosage 
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As the data in Table 30 show, five teachers each were identified in the relatively high and 
low groups, and the middle group was composed of seven teachers. Mean scores on the range of 
implementation variables were then computed for each group and statistical differences between 
groups examined through ANOVA difference in means tests. As with the implementation 
findings reported earlier, results show wide variation within each of the three groups. No 
differences were found to be statistically significant, which is not surprising given the small 
sample size and the substantial within-group variation. Although a highly tentative and 
exploratory finding, it seems noteworthy that the relatively low group shows relatively less 
implementation on nearly all of the variables than does the relatively high group—or stated 
alternatively, while no causality can be attributed, teachers whose students performed relatively 
the best were more thorough implementers than those whose students fared relatively the worst. 


55 


Chapter 6: Summary and Conclusions 


This report has summarized CRESST’s study of the implementation and effects of LDC in 
early-implementing eighth-grade history/social studies and science classrooms in Kentucky and 
Pennsylvania. The study is one of two’ conducted by CRESST, with funding from the Bill and 
Melinda Gates Foundation to examine how LDC supports secondary teachers’ and students’ 
transition to the Common Core State Standards in English language arts. Both studies address the 
following evaluation questions: 

1. How do teachers implement LDC? 

2. What is the impact of LDC on student learning? 

3. What conditions and contexts, including quality of implementation, influence LDC 

effectiveness? 

In the sections below, we consider contextual factors that are important in interpreting 
study results before summarizing our findings with regard to each question. We conclude with 


implications and next steps for research and practice. 
Contextual Considerations 


The nature and generalizability of the study sample present important limitations for the 
study. The study addresses only a subsample of those schools, teachers, and students across the 
country and even within Kentucky and Pennsylvania who currently are implementing LDC. The 
study includes only teachers and students in those districts and schools that were early 
implementers and of these, only those in the targeted subjects and grade level. Because of the 
Foundation’s interest in a rigorous quantitative study, our study design required common 
outcome measures and could not accommodate scores from different grade-level assessments. 
Further, the study focuses on teachers who had at least one year prior experience in 
implementing LDC, so that it would not be judging intervention effects as teachers were initially 
learning how to implement LDC. Thus, the study is limited to districts that were funded in Phase 
1 and Phase 2 of LDC’s initial rollout. 


Even as we attempted to maximize sample size by drawing on sites across two subject 
areas and two states, study power and generalizability are limited. Because of data availability, 
we could conduct a rigorous quasi-experimental design only in Kentucky, where our sample was 
limited to students taught by 37 eighth-grade history/social studies and/or science teachers and 
their carefully matched comparison group. This sample size limits the study’s power to detect 
moderate program effects. Our ability to identify relationships between LDC implementation and 


3A companion study examines the implementation and effects of LDC in a districtwide implementation in Advanced 
Reading (see Herman et al., 2015). 
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outcomes is even more constrained, as only about half of the teachers agreed to participate in the 


implementation components of the study. 


The representativeness of the study sample further limits the generalizability of any study 
findings. Demographically and in prior achievement our Kentucky sample looks similar to the 
state as a whole. However, the study cannot control for unobserved variables that may influence 
student success, and indeed by virtue of their willingness to participate in early LDC trials, study 


districts and schools may well be at least somewhat unique. 


That study teachers had minimum prior experience implementing the intervention is still 
another important contextual consideration. The majority of the teachers in the study had only 
one year of experience implementing the intervention prior to the study year, and in fact that 
prior year included both initial learning and initial implementation. One year is hardly adequate 
time for teachers to meaningfully integrate and become effective with new practices—and the 
literary focus of LDC certainly required substantial changes in practice for both Kentucky and 
Pennsylvania study teachers. That is, LDC—as does the Common Core—requires that content 
teachers take responsibility for teaching literacy, a new responsibility for which they have little 
or no prior training. On the one hand, LDC provides a flexible template to enable middle and 
high school teachers to easily integrate CCSS standards in reading, research, and writing into 
their content area assignments, but on the other hand, the ongoing pedagogy to support their 
students’ literacy development is a new, to-be-learned skill for the great majority of these 


teachers. 


Intervention dosage for both teachers and students is another factor worth consideration. 
Study LDC teachers had implemented only one or two 2—4 week modules prior to the study year. 
Similarly, student dosage—the amount of treatment students received—also was limited. For the 
study year, LDC-oriented instruction made up only four to six weeks of the school year. It is 
ambitious, in short, to expect LDC to have measurable impact student learning at this early point 


in implementation. 
How Did Teachers Implement LDC? 


Twice-weekly teacher logs and end-of-year teacher survey results indicate that LDC study 
teachers did implement the major components of the intervention. They followed the LDC 
framework in developing and using their modules, introducing module content and goals, 
engaging students in reading module texts, transitioning to writing, and working with students on 
their end-of-module writing assignment. The bulk of module time, as would be expected, was 
spent in the reading and writing components. In implementing these components, teachers 


reported developing their students’ skills in a range of reading and writing strategies, although 
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both survey and log results show substantial variability across teachers. Similarly, teachers 
reported frequent use of formative assessment: they reported using a variety of strategies to 
monitor their students’ ongoing learning for both reading and writing, and generally reported 
taking action when misunderstandings and/or problems were observed, again with substantial 
variation in the strategies used. The log and survey data of course are self-report data, from 
which we can better infer the frequency of reported behavior than the quality of that behavior 
(Porter, 2002). We thus cannot directly infer the quality of teacher practice from the log data. 


The study’s analysis of LDC modules provides a more direct window into issues of quality. 
A specially developed assignment measure was used to assess the quality of teacher-developed 
modules on nine dimensions. Ratings by trained expert teachers indicate Kentucky and 
Pennsylvania modules were generally in the middle to high range of moderate quality. The 
relatively highest ratings were for the fidelity to LDC module instruction dimension, which again 
suggests teachers’ commitment to implementing the model. As with other implementation 


findings, however, results showed wide variation in quality ratings across teachers. 


The wide variation in Kentucky modules and in implementation strategies and teacher 
preparation, as captured by teacher logs and surveys, may provide one reason why we could not 
find strong relationships between any single LDC implementation measure and student learning 
outcomes. That is, the quality and effectiveness of LDC implementation depend on the quality of 
the assignments in which students are engaged, as measured through the modules and the ways 
in which those activities are implemented in classroom interaction, as we attempted to detect in 
the log and survey measures, among other unobservables. A high-quality module that is not 
implemented with effective teaching, assessment, and learning strategies would not be expected 
to have a strong effect on student learning, while a poor-quality module that is not consistent 
with important content and literacy goals also would not be expected to have an impact. In other 
words, there may be an interaction between the qualities assessed by our log and survey measure 
and those assessed by the module that we are unable to investigate with our current small 


sample. 
How Did LDC Affect Student Learning? 


Teacher perspectives. The implementation data, while showing wide variation, suggest 
that teachers overall were committed to the LDC intervention. Survey results indicate that 
Kentucky and Pennsylvania teachers found LDC to be a helpful and effective tool in meeting a 
wide variety of instructional goals, including implementing the CCSS, incorporating formative 
assessment and teaching literacy in content area classes, and increasing the rigor of writing 


assessments. Although sample sizes are too small to draw firm inferences, there were indications 
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that science teachers were more challenged by implementation than were social studies or ELA 
teachers: Science teachers reported less involvement in professional development, less 


collaboration with their peers on LDC, and more obstacles to LDC implementation. 


The majority of LDC teachers also agreed that their students experienced at least some 
success in each of the LDC component tasks—the reading mini-tasks, writing mini-tasks and 
final writing task. At the same time, however, Kentucky and Pennsylvania teachers also noted 
that at least some of their students struggled, suggesting that content area teachers and their 
students may need help to increase the productivity of the reading and writing mini-tasks, as well 
as success on the final writing task. Although there was variation across respondents, teachers 
tended to agree at least somewhat that LDC had resulted in higher quality student writing, and 
supported students’ college readiness. 


CRESST ILA results. Students’ performance on the CRESST ILAs stand in some contrast 
to teachers’ positive perspectives but underscore teachers’ concern about LDC’s success with all 
of their students. The CRESST ILA generally parallels the sequence of reading and writing 
activities in LDC: Students are asked to read and respond to several related texts about a central 
subject matter concept or topic and then to synthesize what they have read with their existing 
knowledge to write an extended argumentative or explanatory essay. ILA topics for social 
studies and science were selected to be consistent with eighth-grade content standards in both 
Kentucky and Pennsylvania—Reconstruction for social studies, and evolution for science. 
Student essays were scored on five dimensions: content understanding, rhetorical structure and 
quality, organization, use of evidence/text support, and grammar and conventions. Specially 
trained, expert teachers used a four-point scale to rate each dimension, where a score of 4 
represented advanced performance, 3 represented proficient, 2 represented a basic level of 


performance, and 1 below basic, relative to relevant Common Core State Standards for ELA. 


ILA results for participating eighth-grade LDC social studies and science students were 
disappointing in that students scored between a Level 1 and Level 2 on all five dimensions. 
Performance on the grammar and conventions dimension tended to be the highest, but did not 
reach the level of basic. Limited data from the study’s opportunity to learn survey may provide 
important context here, particularly for the science ILA. The survey asked teachers about their 
students’ curriculum exposure to the given ILA topic. Relatively few teachers responded. 
However, the responses of those who did raise questions about whether students had the 
anticipated prior exposure, despite the topic being part of state grade-level standards for both 


content areas. 
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Even so, because the structure of the ILA provides students relevant, grade-appropriate 
reading and context sufficient to respond to the ILA essay question, the absence of prior 
exposure should not have been a fatal problem. Motivation may have also depressed 
performance as the test was given at the end of the school year. Nonetheless, the results suggest 
the challenge of moving student performance from current status to the expectations of the 
Common Core. 


LDC impact on student learning. Our quasi-experimental design methodology was 
realized only for the LDC study in Kentucky, due to data availability issues described earlier. 
The methodology used Coarsened Exact Matching to identify a group of comparison students 
who were demographically and academically similar to the study LDC students. The matching 
was done at the student level, but accounted not only for student demographics and prior 
achievement, but also for the prior effectiveness of teachers and schools. The resulting treatment 
and comparison student samples were used to test LDC effects on three outcomes: K-PREP 
writing, K-PREP reading, and K-PREP social studies. For each of these outcome measures, two 
separate, two-level hierarchical linear models were run, each modeling students’ dosage under 
treated and non-treated teachers in eighth-grade science and social studies courses, and each 
incorporating measures of teacher effectiveness on the outcome measure of interest prior to the 
LDC intervention as additional value-added controls. Student demographic and _ prior 
achievement variables and school prior effectiveness also were included in the models, as were 
the interactions of these variables with the LDC treatment. Our estimates therefore control for 
observables in two ways, at the matching and modeling stages. 


Results for all three outcomes were consistent for both models, suggesting the robustness 
of these findings. The analysis found no evidence of an LDC effect for writing or for social 
studies. However, for K-PREP reading, both models showed a statistically significant, positive 
effect for LDC. LDC teachers showed a positive value added relative to control teachers. Results 
suggest that having one LDC teacher improves a student’s performance on the K-PREP reading 
assessment by 0.058 standard deviation points, corresponding to 0.91 scale score points. The 
effect of having both a social studies and science LDC teacher, based on these data, would be a 
1.82 scale point increase in reading scale score. These results thus suggest that LDC’s attention 
to reading in content-area classrooms is contributing to the students’ reading performance— 
decidedly positive news for the intervention. We return to these observed effects in our 


conclusion. 
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What Conditions and Contexts Influence LDC Effectiveness? 


Interesting interaction effects emerged from our QED analysis that point to conditions and 
context that influence LDC implementation and impact. In particular, both students’ prior 
achievement, based on their prior year K-PREP scores, and students’ socioeconomic status 
(SES), as revealed by their free or reduced price lunch status, show positive interactions with the 
treatment. That is, LDC students who were relatively higher achieving prior to their LDC 
experience showed relatively greater benefit than did those who started relatively lower 
achieving, although the observed effect is very small. Interestingly, LDC students receiving free 
or reduced price lunch also appeared to have benefited more from LDC, after controlling for 
other variables. Although, again, the observed effect was very small, we speculate that LDC 
students with lower SES status perhaps had access to special resources (e.g., Title I programs, 
specialist teachers) that provided essential support. Controlling for other factors, special 
education students appeared to do less well under LDC; however the share of students falling 
into this category was small. 


The results for the K-PREP social studies analysis also revealed similar interaction effects. 
LDC students who started the year performing at a relatively higher level experienced more 
benefit from LDC in their social studies performance, as did students who were from a relatively 
lower SES, as evidenced by their free or reduced price lunch status. However, the study did not 
find differential treatment effects for special education students. 


Conclusions 


In summary, LDC shows promising, positive results in supporting teachers’ transition to 
the College and Career Ready Standards (CCRS), and in improving student learning. At the same 
time, however, study findings suggest challenges that LDC will need to overcome to move to 


higher levels of success. We summarize our perspective on major study implications. 


Positive effects on student learning. That LDC shows statistically significant results on 
Kentucky students’ state assessment scores in reading is worth celebrating. This positive finding 
is particularly so in light of both study teachers’ limited prior experience implementing the tools 
and the limited dosage students experienced. That is, as noted earlier in this chapter, study 
teachers had only one or two years of experience with LDC prior to the study year, and for the 
great majority it was only one year. Based on research on teachers’ implementation of new 
practices, this is insufficient time for teachers to become fully comfortable and competent with 
the kinds of new pedagogical practices that LDC represents (Coburn, 2003; Hargreaves & 
Fullan, 2012). Consider that LDC requires that content teachers take responsibility for teaching 
literacy, a new responsibility for which they have had little or no prior training. 
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Intervention dosage is another factor to consider in evaluating LDC effects. In general, the 
longer and more intensive the treatment, the more likely an intervention is to show measurable 
effects. LDC teachers typically implemented two modules of two to three weeks’ duration each 
during the study year, meaning that LDC-oriented coursework totaled only four to six weeks, 


only a small fraction of the full academic year. 


Nonetheless, the study found a statistically significant learning effect for LDC in Kentucky, 
approximately equivalent to 2.2 months of regular schooling. Given their contexts of early 


implementation and limited dosage, this effect is noteworthy. 


Positive effects on teachers. The effect found for student learning is matched by teacher 
enthusiasm for the tool. Across states and districts, teachers were positive about the professional 
development they received and reported that they found the tools helpful and effective in 
meeting a variety of goals, including implementing CCRS, using formative assessment, 
incorporating more complex thinking and problem solving into curriculum and instruction, and 
improving student learning. Teachers’ reports about their fidelity of tool implementation provide 


additional evidence of their positive attitudes. 


Struggles in moving to higher standards. While our study found positive effects on 
teachers and students, findings also demonstrated the challenge of moving to more rigorous 
Common Core State Standards. We see evidence of this challenge in students’ low performance 
on measures specifically designed to reflect the deeper learning demands of new college and 
career ready standards and in teachers’ reports that sizable proportions of their students are 
struggling relative to the goals of LDC. Our analysis of LDC classroom artifacts also indicate 
that some teachers struggled in their implementation efforts, as would be expected given this 


early stage of implementation. 


That some teachers and students struggled is not meant to imply that current standards are 
unattainable or that college and career ready expectations for students should be reduced—after 
all, we know that returning to prior standards will not get our children to 21st century success. 
However, the evidence does suggest that change will not come overnight and that both teachers 
and students will need support to meet the challenge. The issue is two-fold: (1) How to address 
the needs and better prepare students and teachers who may not yet be ready to be successful 
with the challenges of LDC; and (2) how to modify and/or adapt the tools to scaffold teacher and 


student learning more effectively. 


Achievement gap implications. Although we regard findings of the interaction between 
student characteristics and treatment effects as tentative and subject to further validation, the 


consistency in results across reading and social studies measures is striking. While the overall 
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results indicated that LDC was effective for all Kentucky students, the interaction findings 
indicated that initially higher achieving students benefited more than did initially lower 
achieving students. Such a finding makes intuitive sense in that lower achieving students have 
most likely been exposed to the “drill and kill” test preparation curriculum of the past, are least 
likely to have acquired the prior grade knowledge and skills expected by the Kentucky Core, and 
are least likely to have been engaged in the deeper conceptual understanding and applications 
that mark the new standards. 


However understandable, the findings thus suggest that, in the absence of additional 
scaffolding and supports for low-achieving students, LDC is likely to be gap enhancing. On the 
other hand, study findings of a positive interaction between LDC and students’ free and reduced 
price lunch status offers promise for future inquiry. The results suggest that, controlling for prior 
achievement and other background characteristics, students who are more economically 
disadvantaged fare relatively better under LDC than their more advantaged peers. 


Strengthening implementation. Although teachers reported implementing all components 
of both LDC, the findings suggest substantial variation in how they implemented the tool and in 
the relative time and specific strategies they used in doing so. The study did not achieve strong 
findings with regard to what aspects of implementation mattered most or what specific strategies 
were most effective. The findings are suggestive, however, of some factors that might be 
important for success: District support for LDC was clear across the sample, yet principal or 
local school support was more variable, suggesting a potential problem point. Teachers found 
their peers highly collaborative and helpful in implementing the two tools, but time for 
collaboration and more formal professional development was somewhat limited; investing more 


heavily in these supports may strengthen implementation. 


Concluding thoughts. In summary, our studies reveal that study teachers are enthusiastic 
about LDC, and that LDC showed important effects on student learning. Even so, study results 
also suggest areas for improvement. Content teachers who implement LDC likely will be more 
successful to the extent they have expertise in supporting students’ literacy development. 
Additional supports for struggling students, and training on how to successfully implement LDC 
with students with special needs could potentially help close achievement gaps. 


We leave it to future research to examine the generalizability of these findings in the larger 
samples of teachers and schools that are now implementing LDC. Cost-effectiveness studies also 
should be of interest. Future research and development also should continue the quest to identify 


both the most critical aspects of implementation in improving student learning and key 
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infrastructure and supports that students and teachers who currently are struggling need to propel 


their success. 
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Appendix A: 
LDC and MDC Instruments and Rubrics 


Exhibit Al: 


Kentucky and Pennsylvania LDC Teacher Log 


Thank you for taking the time to complete your teacher log for LDC instruction. This log 


should take no longer than five (5) minutes to complete. 


As a reminder, your logs will track instruction in only one of your LDC classes. Please 
remember to focus on the class period that was assigned to you when completing this log. 


For each log you complete, describe what happened in that particular class on that day. 


If you have not yet completed module instruction, click "Continue with log." If you have 
completed module instruction and are returning to upload files, check “Skip log to upload 
files” below. 

© _ Skip log to upload student work 


© Continue with log 


Is this your first log for this module? 
OQ Yes 
OQ No 


Is this your FINAL LOG for the current LDC module? 


OQ Yes 
OQ No 
© Not sure 


What are your primary goals for this module? 


Content goals 


Reading goals: 


Writing goals: 


How much total class time did students spend on any part of an LDC module today? 


0 minutes < 10 minutes 10-30 minutes 30-50 minutes >50 minutes 


O O O O O 


Why was there no class time devoted to LDC today? 


I was absent 


School not in session 


Need to cover other content 


Field trip 


Other 


Check which aspects of the module students focused on today. (Check all that apply). If you 
are not sure how to characterize today’s lesson, check the aspect that best describes the focus 


of instruction. 


Preparing for the Task/Introducing Module 


Reading Process 


Transition to Writing 


Writing Process 


Consider the total amount of class time spent on module instruction today. What proportion 


of time was spent on the following classroom activities? (Total must add up to 100%. If this 


list does not include certain classroom activities, please list those in the text box under 


“Other” below). 


Lecture on subject matter content 
Mini-lesson on 


Explicit strategy instruction (e.g., 
teacher think-aloud, modeling, 
guided practice about a specific 
strategy) 


Whole-class discussion 
Small group work 
Pair/share 

Independent reading/writing 
Student presentations 


Other 


80 90 100 


Preparing for the Task/Introducing Module 


How did you introduce the module? (check all that apply) 


Focus of 
instruction 


Touched on 
briefly 


Not today 


Overview and/or review of topic 


Connect topic to students’ existing knowledge 


Overview of readings 


Review writing prompt 


Review success criteria and/or rubric 


Other 


O 


YO ar) 


O 


Cy er CEO 


Ke? NR ED 


What did students do to prepare for module instruction? (check all that apply) 


Focus of Touched on Not today 
student work briefly 
Listened as I explained task O O O 
Came up with questions about topic O O O 
Made predictions about topic '@ O O 
Came up with questions about writing task O O O 
Re-wrote task in their own words O O © 
Brainstormed possible answers to prompt O O O 
Made predictions about reading O O O 
Reviewed exemplars of student work O O O 
Completed planning sheet (e.g., graphic organizer) O O O 
Discussed important strategies needed to complete task O O O 


Reading Process 


Which reading skills did students work on today? (check all that apply) 


Focus of Touched on Not today 
student work briefly 
Independent reading research O O O 
Making predictions/previewing O O O 
Summarizing important points O O O 
Note-taking/annotation © O O 
Identifying/ defining vocabulary O O O 
Analyzing text structure (e.g., how part relates to whole) O O O 
Interpreting information from graphical text O O '@) 
Distinguishing fact from opinion O O O 
Drawing conclusions from textual evidence O '@) O 
Citing textual evidence to support claims O O O 
Evaluating strength/weakness of evidence O O O 
Comparing arguments in two or more texts Cc O O 
Examining author's perspective/bias O O O 
Examining rhetorical devices © O O 
Other © O O 


Did you assess student learning of the skills listed above? 


OQ Yes 
OQ No 


If YES, to what extent did you rely on the following strategies to assess student 


understanding? (check all that apply) 


To a great To some Not at all 
extent extent 

Listened as students discussed text with peers O O O 
Circulated and reviewed student notes O O O 
Reviewed peers' feedback O O O 
Collected and reviewed student written responses and/or O O O 
graphic organizers 

Asked students to answer oral questions O O O 
Listened to students thinking aloud while reading O O O 
Led whole-class discussion O O O 
Listened to student questions Ss O O 
Assigned a quiz O O O 
Graded student work O O O 
Exit slips © O O 


If you did discover student misunderstanding about READING, to what extent did you rely 
on the following strategies to respond? (check all that apply) 


To a great To some Not at all 
extent extent 

One-on-one conference to provide feedback O O O 
Asked peer to provide feedback O O O 
Stopped class and modeled strategy O O O 
Wrote specific comments on student work O O © 
Scheduled in-class workshop time O O O 
Devoted time in lesson for students to use feedback O Ce) O 
Grouped students together on a "need" basis for targeted O O © 
instruction 

Offered student a hint or suggestion O e) O 
Gave student the answer O O O 
Gave student more time to try again and self-correct O O O 
Graded student work © O O 
Re-taught lesson segment O O O 
Planned to review skill in future lessons O O O 
Other O O O 


Transition to Writing 


How did you help students prepare for the writing task? (check all that apply) 


Focus of Touched on Not today 
instruction briefly 
Overview and/or review of topic O O O 
Review of readings O O O 
Review writing prompt O O O 
Review success criteria and/or rubric O O O 
Other O O O 


What did students do to prepare for the writing task? (check all that apply) 


Focus of 
instruction 


Touched on 
briefly 


Not today 


Listened as I explained task 

Came up with questions about topic 

Made predictions about topic 

Came up with questions about writing task 
Re-wrote task in their own words 
Brainstormed possible answers to prompt 
Generated thesis statements 

Reviewed exemplars of student work 
Completed planning sheet (e.g., graphic organizer) 
Generated essay outline 

Selected relevant quotes from documents 


Discussed important strategies needed to complete task 


O 


GE I HE eS A: A 


O 


CC Cm 6G Ome ae mo me oe 


Ce OG WE) CA ED I’ A a a SD 


Writing Process 


What areas of writing did students work on today? (check all that apply) 


Focus of 
student work 


Touched on 
briefly 


Not today 


Generating ideas for writing 
Outlining 

Writing/text structure 
Formulating a thesis statement 
Formulating a counter-argument 
Writing an introduction 

Writing a conclusion 

Writing a body paragraph 

Using transitional words or phrases 
Incorporating quotes/evidence 
Style/word choice/syntax 


Grammar conventions 


O 


Ce On ee HO a Be 


O 


Ce I OS SO A Hy SED Me) 


CE I A) 


Did you review student learning of the skills listed above? 


OQ Yes 
OQ No 


If YES, to what extent did you rely on the following strategies to assess student 


understanding? (check all that apply) 


To a great To some Not at all 
extent extent 
Listened as students discussed draft with peers O O O 
Asked students to provide feedback to each other O O O 
Observed and reviewed student work O O O 
Collected and reviewed student writing exercises O O O 
Asked students to answer oral questions O O O 
Reviewed student rough drafts O e@ O 
Asked certain students to present writing to class O O O 
Assigned a quiz O O O 
Graded student work O O O 
Exit slips © O O 


If you did discover student misunderstanding about WRITING, to what extent did you rely 
on the following strategies to respond? (check all that apply) 


To a great To some Not at all 
extent extent 

Organized peer-editing session O O O 
Scheduled in-class workshop time O O O 
Held one-on-one conference with student O O O 
Devoted time in lesson for students to use feedback O O O 
Grouped students together on "need" basis for targeted O O O 
instruction 

Modeled skill using my own writing O O ©) 
Demonstrated skill using student's writing O O 2) 
Provided grammar mini-lessons O O O 
Wrote specific comments on student work O O O 
Had student revisit readings O O O 
Offered student a hint or suggestion O O O 
Gave student time to try again and self-correct O O O 
Corrected student writing O O O 
Graded student work © O O 
Re-taught lesson segment O O ‘@ 
Planned to review skill in future lessons O O O 
Other © O O 


You have indicated that this will be the last week of module instruction. If that is the case, we 


have a few final questions about the particular module you taught. 


Module Topic 


LDC template (number, type, level) 


Writing prompt 


How many weeks did you spend teaching this LDC module? 


How many days of instruction for each of the following? 


Preparation for the task 


Reading process 


Transition to writing 


Writing process 


Please list the approximate start dates of your next two modules. 


Module Two 


Module Three 


To understand how you approach LDC instruction in your classroom, we would like you to 
submit some classroom materials, including your LDC module template, copies of the 
readings/texts you assigned as part of LDC instruction, three samples of student work, and 
instructional materials used to support reading and writing instruction. You can attach PDFs 
or Word files. If you would prefer to send us hard copies, please email Scott Epstein at 
epstein@cse.ucla.edu. If you would like to upload files later, you can return to the log and 


click "skip to upload files" on the first page. 


Please upload a filled out LDC module template, including the teaching task, skill list, mini- 
tasks, and instructional strategies for the module. (If you did not use the LDC template, 


please upload an outline of your LDC instruction). 


Browse 


Please scan and upload a full set of readings/texts that were assigned to students or that 
students read as part of module instruction. If your students selected their own reading 


materials, please include two texts that are representative of what students read. 


Browse 


Please scan and upload three samples of student work on the template task. Please select one 
student who performed at a high level on the template task, one student who performed at a 
medium level, and one student who performed at a low level. Please label the 3 sets of work 


as high, medium, and low, and if possible remove any _ student names. 


Sample of high student work: 


Browse 


Sample of medium student work: 


Browse 


Sample of low student work: 


Browse 


Please scan and upload instructional materials you used during the module (e.g., worksheets, 
handouts, graphic organizers, etc.) that would help us better understand what students 
actually did during classroom instruction. Please include at least one item supporting reading 


instruction, and one item supporting writing instruction. 


At least one example of instructional materials supporting reading: 


Browse 


At least one example of instructional materials supporting writing: 


Browse 


Any other planning or instructional materials that would help us understand how LDC was 


implemented in your classroom: 


Browse 


ADDITIONAL COMMENTS: 


We are very interested in your feedback. Please let us know if you have any questions or 
concerns about this log. Thank you! 


Exhibit A2: 
Rubric for LDC Module Implementation Measure 


Each dimension is scored on a 5-point scale ranging from “Fully Present or Realized” to “Not Present 
or Realized.” 


Fully Sufficiently Moderately Present Barely Not 
Present or Present or or Present or Present or 
Realized Realized Realized Realized Realized 
5 4 3 2 1 


IMPORTANT: Descriptions are provided for three anchor points in the scale: 5 (Fully Present or 
Realized), 3 (Moderately Present or Realized), and 1 (Not Present or Realized). Use the intermediate 
points in the scale (4 and 2) to rate assessment practice that lies between 5 and 3 and 3 and 1. 


Dimension 1: Effective Writing Task 

Dimension 2: Alignment to the CCSS and Local and State Literacy and Content Standards 
Dimension 3: Text Alignment 

Dimension 4: Text Appropriateness 

Dimension 5: Text Rigor 

Dimension 6: Fidelity to LDC Module Instruction 

Dimension 7: Quality Instructional Strategies 


Dimension 8: Coherence and Clarity of Module 


Dimension 9: Overall Impression 


Definition: Degree to which teaching task makes effective use of the template task’s writing mode 
(i.e., argumentation or explanation); requires sustained writing and effective use of ideas and 
evidence to substantiate claims; and is feasible for most students to complete (i.e., appropriate for 
the grade-level and subject matter). 


Main Sources of Information: 
Module Creator Handout (Task) 


- Read and evaluate the teaching task, student background/prior knowledge, and summary 
information. 


- Evaluate the difficulty or ease students may encounter trying to answer the question. 
- Compare module teaching task to teaching task template options. 


5. Fully Realized The teaching task and performance expectations for the 
module are explicit and clear, require students to engage in 
higher-order thinking and writing, and are appropriate for the 
grade-level and subject matter. 


4. 


3. Moderately Present or Realized Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher- 
order thinking and writing and/or are not appropriate for the 
grade-level and subject matter. 


2. 


1. Not Present or Realized Minimal evidence of an effort to identify explicit and clear 
teaching task and performance expectations that provide 
opportunities for critical thinking and are appropriate for the 
grade-level and subject matter. 


Definition: Extent to which module addresses content essential to the discipline, as well as reading 
comprehension and writing standards informed by local and state standards. 


Main Sources of Information: 
Module Creator Handout (Task) 


- Read and evaluate the standards included in the module. 


- Module should include ELA as well as subject matter CCSS/state standards. 


- Compare and contrast the standards the module includes with those that could have been 


included. 


- Particular attention to content standards (CCSS History/Social Studies, Science, and Technical 
Subjects); State Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 


5. Fully Realized 


Module specifically addresses content essential to CCSS and 
local or state standards in science or social studies, as well as 
reading comprehension and writing. All standards are well 
aligned to the topic and teaching task. 


4. 


3. Moderately Present or Realized 


Module broadly addresses content essential to CCSS and local 
or state standards in science or social studies and reading 
comprehension and writing. Standards are sufficiently aligned 
to the topic and teaching task. 


2. 


1. Not Present or Realized 


Minimal evidence that module addresses content essential to 
the discipline and literacy standards. Standards are poorly 
aligned to the topic and teaching task. 


Definition: Degree to which assigned texts address teaching task content. 


Main Sources of Information: 


Student Work 
- References in student work. 


Module Creator Handout (Task, Resources, Links) 
- Read and evaluate texts (hard copies or online). 


5. Fully Realized 


Assigned readings address the disciplinary content in science 
or social studies and give students the opportunity to gather 
information needed to complete the task. Readings are well 
aligned to the topic and teaching task, and provide students 
with well-balanced perspectives. 


4. 


3. Moderately Present or Realized 


Assigned readings mostly address the disciplinary content in 
science or social studies and give students some opportunities 
to gather information needed to complete the task. Readings 
are sufficiently aligned to the topic and teaching task, and 
provide students with moderately balanced perspectives. 


2. 


1. Not Present or Realized 


Minimal evidence that assigned readings address the 
disciplinary content in science or social studies and give 
students the opportunity to gather information needed to 
complete the task. Readings are poorly aligned to the topic 
and teaching task, and do not provide students with well- 
balanced perspectives. 


Definition: Degree to which teaching task includes reading texts that are accessible to most students 
(i.e., appropriate for the grade-level and subject matter). 


Main Sources of Information: 


Module Creator Handout (Task, Resources, Links) 


- Read and evaluate texts (hard copies or online). 


Student Work 
- References in student work. 


Anchor Readings 


- Read for examples of appropriate reading levels for gi grade students. 


5. Fully Realized 


Assigned readings are highly accessible and appropriate for 
most students in 8" grade social studies or science classrooms. 
Selection of readings addresses the needs of students with a 
range of literacy skills, including students who are above, at, 
or below grade level, and English Language Learners. 


4. 


3. Moderately Present or Realized 


Assigned readings are mostly accessible and appropriate for 
the majority of students in 8" grade social studies or science 
classrooms. Selection of readings sufficiently addresses the 
needs of students with a range of literacy skills. 


2. 


1. Not Present or Realized 


Assigned readings are not accessible or appropriate for 
students in 8" grade social studies or science classrooms. 
Selection of readings poorly addresses the needs of students 
with a range of literacy skills. 


Definition: Degree to which teaching task includes reading texts that use and develop academic 
understanding and vocabulary, and offer opportunities for multiple interpretations and higher-order 
thinking. 


Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 
- Identify list of selected articles/links. 

- Read and evaluate texts (hard copies or online). 
- Consider issues of source credibility. 

Student Work 

- References in student work. 


5. Fully Realized Assigned readings require students to engage in higher-order 
thinking, and develop a strong academic understanding and 
vocabulary in social studies or science. Readings afford a deep 
conceptual and contextual understanding of the teaching task 
and topic. Selection of readings includes a broad range of 
credible primary and secondary sources. 


4. 


3. Moderately Present or Realized | Assigned readings require students to engage in some higher- 
order thinking, and develop an adequate academic 
understanding and vocabulary in social studies or science. 
Readings afford a sufficient conceptual and contextual 
understanding of the teaching task and topic. Selection of 
readings includes a moderate range of credible primary and 
secondary sources. 


2. 


1. Not Present or Realized Assigned readings require students to engage in little higher- 
order thinking, or develop an academic understanding and 
vocabulary in social studies or science. Readings afford a 
limited conceptual and contextual understanding of the 
teaching task and topic. Selection of readings includes few 
credible primary and secondary sources. 


Definition: Degree to which module instruction, activities, and teaching task address each of the four 
stages of instructional practice (preparation for the task, reading process, transition to writing, 
writing process). 


Main Sources of Information: 
Module Creator Handout (Instruction) 
Information Sheet 


- Evaluate for distribution of activities and time spent on each of the four stages of instructional 
practice. 


5. Fully Realized The module instruction, activities, and teaching task reflect 
deliberate attention and fidelity to the four discrete stages of 
LDC module instruction. Classroom materials reflect 
demonstrable effort to develop instructional scaffolding within 
and across each stage of instruction. 


4. 


3. Moderately Present or Realized The module instruction, activities, and teaching task reflect 
moderate attention and fidelity to the four discrete stages of 
LDC module instruction. Classroom materials reflect sufficient 
effort to develop instructional scaffolding within and across 
each stage of instruction. 


2. 


1. Not Present or Realized The module instruction, activities, and teaching task reflect 
poor attention and lack of fidelity to the four discrete stages of 
LDC module instruction. Classroom materials reflect 
inadequate effort to develop instructional scaffolding within 
and across each stage of instruction. 


Definition: Degree to which the module provides clear instructional strategies aimed at helping 
students develop literacy skills and successfully complete the teaching task. And the degree to which 
module instruction and activities scaffold critical thinking and performance in a way that is 
meaningful within the context of a given field or subject-matter. 


Main Sources of Information: 

Module Creator Handout (Instruction) 

Classroom Handouts 

Student Work 

- Evaluate extent to which instructional strategies guide student learning in literacy and ability to 
complete the teaching task. 

- Evaluate extent to which the module activities scaffold critical thinking and student performance 
within the context of the subject matter at the core of the teaching task. 


5. Fully Realized Module provides clear and targeted instructional strategies 
and activities that scaffold student learning and promote 
critical thinking in social studies or science. There is explicit 
attention to helping students develop an accurate 
understanding of the topic and teaching task, and literacy 
skills necessary to successfully complete the writing task. 


4. 


3. Moderately Present or Realized Instructional strategies and activities are available to support 
adequate student learning and critical thinking in social 
studies or science. There is moderate attention to helping 
students develop an understanding of the topic and teaching 
task, and literacy skills necessary to complete the writing task. 


2. 


1. Not Present or Realized Limited instructional strategies and activities available to 
support student learning and critical thinking in social studies 
or science. Insufficient attention to helping students develop 
an understanding of the topic and teaching task, or literacy 
skills necessary to complete the writing task. 


Definition: Degree to which there is logical alignment between the teaching task and other module 
goals with readings, mini-tasks, and instructional strategies. 


Main Sources of Information: 
Module Creator Handout 
Classroom Handouts 
Student Work 


5. Fully Realized 


Strong alignment between the teaching task and goals of the 
module, including the CCSS and local and state literacy and 
content standards, with the readings, mini-tasks, student 
work, and instructional strategies. 


4. 


3. Moderately Present or Realized 


Moderate alignment between the teaching task and goals of 
the module, including the CCSS and local and state literacy and 
content standards, with the readings, mini-tasks, student 
work, and instructional strategies. 


2. 


1. Not Present or Realized 


Poor alignment between the teaching task and goals of the 
module, including the CCSS and local and state literacy and 
content standards, with the readings, mini-tasks, student 
work, and instructional strategies. 


Definition: Holistic assessment of LDC Module. 


Main Sources of Information: 
Module Creator Handout 
Classroom Handouts 
Student Work 


Main question: To what extent does this module contribute to student college readiness and 
development of advanced literacy skills? 


5. Advanced LDC Module Implementation 


4. Proficient LDC Module Implementation 


3. Adequate LDC Module Implementation 


2. Marginal LDC Module Implementation 


1. Inadequate LDC Module Implementation 


Exhibit A3: 
LDC Teacher Survey 2013 


Before you begin, note that the Literacy Design Collaborative (LDC) Initiative goes by a number of different names. 
We use the phrases “LDC framework” or modules to refer to the tools that are part of this initiative. 


Some of the questions in the survey make reference to the Common Core State Standards (CCSS). In different states, this could be referred to 
differently, for example, in the state of Colorado, it is referred to as Colorado Academic Standards. 


The LDC Initiative is funded by the Gates Foundation. 


You are about to enter the survey. To go back a page, please use 


the survey’s red “Back” button, not your browser’s back button. 


Your answers will be saved each time you click “Next.” 


The survey takes about 30 minutes to complete. You may leave and return multiple times. 


If you do return, after entering your login code, you will be placed in the screen you last visited. 


Please select the best answer for each question. Some instructions are in italics. 


TEACHER BACKGROUND INFORMATION 


1. What is / are your current position(s)? Please CHECK ALL that apply. 
C1, Classroom teacher 
TF, Reading specialist 
LC, Reading coach 
Fy Special education teacher 
O, Librarian 
1; Department head 


C1, Other (please specify) [100 characters] 


2. At which grade level(s) do you teach? Please CHECK ALL that apply. 
O1, Middle school (6 - 8" grade) 
O, High school (9 - 12" grade) 


3. Which content areas do you teach? Please CHECK ALL that apply. 
LO, English/Language Arts 
Ol, Science 
C1. Social Studies 
Fy Reading 
O. Other (please specify) 


[100 characters ]_ 


4. To the nearest year, how long have you... 


a) ... been a teacher? year(s) [ integer, 0-99 ] 
b) ... taught in your current school? year(s) [integer, 0-99 ] 
c) ... taught in your current district? year(s) [integer, 0-99 ] 


Yes No 
5. Do you teach ELL students? O, Oo 
6. Do you teach special education students? 
7. Do you teach students who read or write below grade level? 


8. Do you teach students with advanced literacy levels? 


PARTICIPATION IN LDC INITIATIVE 


9. How would you describe your participation in the LDC initiative? 
O, Required 
QO, Voluntary 
Q3 | have not taught a module in 2012-2013. 


QO, | opted out of participating in 2012-2013 (please specify a reason for opting out) [1000 characters] 


10. Is this your first year in the LDC initiative? 
O, Yes Oo No 


11. My involvement with the LDC Initiative has included the following activities: (please CHECK ALL that apply) 
TL, Teaching a teaching task without a full module 
DO, Revising LDC modules that | did not develop myself 
C11, Developing LDC modules 
Kg Teaching LDC modules 
TF. Coaching others on how to use LDC modules 
1; Presenting at an LDC professional development session 


MODULE DEVELOPMENT 


12. How many modules have you developed during the current school year (2012-13)? ee | integer, 0-99 ]___ module(s) 
Please enter a 0 if you have not developed any modules during the current school year (2012-13). 


13. During the current school year (2012-13), | have ... 
O, ... often O, ... sometimes O; ... never [skip if 12=0] 


..developed modules with the support of a colleague. 


14. How many modules have you revised during the current school year (2012-13)? [integer,O-99]___ module(s) 


Please include modules you developed in a previous year AND modules others developed. Enter a 0 if you have not revised any modules 
during the current school year (2012-13). 


15. During the current school year, | have ... 
O, ... often QO, ... sometimes O, ... never [skip if 14=0] 


.. revised modules with the support of a colleague. 


16. How many modules did you teach last year (2011-12)? [integer, 0-99 ] module(s) 
Please enter a 0 if you did not teach any modules last year (2011-12). 


17. How many modules in total will you have taught during the current school year (2012-13)? [integer, 0-99 ] module(s) 


18. During the current school year (2012-13), | have ... 
O, ... often QO, ... sometimes QO; ... never [skip if 17=0] 


..taught modules with the support of a colleague. 


19. Please indicate the frequency with which you are accessing existing modules on online? 
O, ... often QO, ... sometimes O3 ... never 


20. Indicate whether the following people visited your classroom when you were teaching a module: 


Visited Did not visit 
a. District or network LDC project lead Oi Oo 
b. Principal O O 
c. Instructional coach/department head O O 
d. Teacher colleague O O 


Q21 is about teaching literacy. 


21. Please indicate the degree to which you agree or disagree with the following statements: 


Agree Disagree 
Agree Somewhat Somewhat Disagree 


a. Teachers from all content areas should help students improve their reading and writing skills. O, Oz O3 On 
b. Science and social studies teachers do not have time to teach reading and writing. O OQ Oo Oo 


c. Writing assignments can help my students develop a deeper understanding of important 
concepts. O O O O 


PURPOSE OF INITIATIVE 


22. Please indicate the degree to which you agree or disagree with the statements below: 
Agree Disagree 
Agree Somewhat Somewhat Disagree 
The LDC framework is effective in ... 


a. ... improving students’ literacy skills. O, O, O3 On 
b. ... providing a curricular resource for teachers to address the Common Core State Standards. O O O O 
Cc. ... encouraging science and social studies teachers to teach literacy skills. O O O O 
d._... encouraging secondary school teachers to teach literacy skills. O O O O 
e._ ... making instruction more engaging for the students. O O O O 
f. _... using formative assessment to identify student strengths and weaknesses to inform 


O 
O 
O 


instruction. O 


TEACHER PERCEPTIONS OF TOOL UTILITY 


23. Please indicate whether using the modules has helped you in the following ways during module instruction: 


During LDC instruction, using the modules has helped me... 


a. 


b. 


.. find effective strategies for teaching my subject content. 

.. learn new ways to include formative assessment in my classes. 

.. develop new ways to teach literacy skills in my content area. 

.. learn detailed information about my students’ literacy strengths and weaknesses. 
.. provide students with more detailed feedback about their writing. 

.. implement the Common Core State Standards. 

.. increase the rigor of writing assignments. 


.. better engage students. 


Yes 


O1 


000000 0 


000000 0 


24. Please indicate the degree to which you agree or disagree with the statements below. 


Agree 
LDC modules help me differentiate instruction ... 

[skip next if Q5 = no] 

a. ... for ELL students. O1 
[ skip next if Q6 = no] 

b._... for special education students. O 
[ skip next if Q7 = no] 

c. ... for students who read or write below grade. O 
[ skip next if Q8 = no] 

d._... for students with advanced literacy levels. O 


Agree Disagree 
Somewhat Somewhat Disagree 


Oz O3 On 
O O O 
O O O 


25. The modules are flexible enough to fit the needs of all my students. O O O O 


Questions 26a-d are about the most recent LDC module you taught. 


26. Please indicate the degree to which you agree or disagree with the statements below. 
Agree Disagree 
Agree Somewhat Somewhat Disagree 


a. | knew what skills my students needed in order to complete the teaching task. O, O, O3 Oy 


b. | knew the type of mini-tasks to give my students to prepare them to complete the 
template task. O O O O 


c. |understood how to use the LDC instructional ladder. O O O O 


d. Based on the information collected from using the LDC modules, | adjusted my 
instruction to meet the needs of individual students. O O O O 


27. Please indicate the degree to which you agree or disagree with the statements below. 


a. Using the modules raised my expectations for students’ writing. O O O O 


b. The LDC framework has become an important part of my instructional practice. O O O O 


28. Select the phrase that best completes the following sentences: 
| use module instructional strategies ... 
O, ... often QO, ... sometimes QO ... rarely 


..during non-LDC instruction. 


STUDENT IMPACT 


29. Compared to my usual instruction, during the use of the modules, my students ... 


O, ...are more engaged. OQ, ...show the same level of engagement. QO, ...are less engaged. 


30. Please indicate the degree to which you agree or disagree with the statements below. 
Agree Disagree 
Agree Somewhat Somewhat Disagree 


a. The modules have resulted in higher quality student writing. O1 Oz O3 On 


b. The LDC framework is supporting my students’ college-readiness. O O O O 


Question 31a-b asks you to reflect back on your most recent experience implementing an LDC module during the current school year (2012-13). 


31a. When | taught the most recent LDC module, the majority of my students improved their understanding of content. 


O, Yes Oo No 


31b. When | taught the most recent LDC module, the majority of my students improved their literacy skills. 
O, Yes Oo No 


Q32-37 are about your LDC instruction in the current school year (2012-13). 


32. What percent of class time did you spend on each of the following LDC components during your LDC instruction in the current school year 
(2012-13)? 


Preparation for Task/Introducing the Module 
Reading Process 

Transition to Writing 

Writing Process 


aoo0w 


33. Please indicate the degree of emphasis you placed on each of the following skills in your LDC reading instruction: 


A great deal of emphasis Someemphasis Little Emphasis No Emphasis 
a. Independent reading/ research O, O, O3 Oy 
b. Making predictions/previewing 
c. Summarizing important points 
d. Note-taking/ annotation 
e. Identifying/ defining vocabulary 
f. Analyzing text structure (e.g. how part relates to whole) 
g. Interpreting information from graphical text 
h. Distinguishing fact from opinion 
i. Drawing conclusions from textual evidence 
j. Citing textual evidence to support claims 
k. Evaluating strength/ weakness of evidence 
|. Comparing arguments in two or more texts 


m. Examining author’s perspective/bias 


0000009000000 0 
0000009000000 0 
000000000000 0 
0000009000000 0 


n. Examining rhetorical devices 


34. Please indicate the degree of emphasis you placed on each of the following skills in your LDC writing instruction: 


A great deal of emphasis Some emphasis Little emphasis No emphasis 
a. Generating ideas for writing O, O, O3 On, 
b. Outlining O O O O 
c. Writing/text structure O O O O 
d. Formulating a thesis statement O O O O 
e. Formulating a counter-argument O O O O 
f. Writing an introduction O O O O 
g. Writing a conclusion O O O O 
h. Writing a body paragraph O O O O 
i. Using transitional words or phrases O O O O 
j. Incorporating quotes/ evidence O O O O 


35. Please indicate how frequently you use the following strategies to assess student learning during your LDC instruction. [ note: randomize 


Often Sometimes Rarely Never 


Listened as students discussed reading or writing with peers 
Asked students to provide feedback to each other 
Circulated and reviewed student notes and work 

Collected and reviewed student writing exercises 

Asked students to answer oral questions 

Reviewed student rough drafts 

Asked certain students to present writing to class 

Assigned a quiz 

Graded student work 

Exit slips 


an 
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w 
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36. Please indicate how frequently you use the following strategies to provide feedback to students during your LDC instruction. 


Often Sometimes Rarely 


Held one-on-one conference with student 

Asked peer to provide feedback or organized peer editing session 
Stopped class and modeled strategy 

Scheduled in-class workshop time 

Wrote specific comments on student work 

Offered student a hint or suggestion 

Gave student more time to try again and self-correct 
Gave student the answer 

Graded student work 

Re-taught lesson segment 

Planned to review skill in later lessons 

Assigned grammar exercises 
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37. Please indicate the degree of success your students had in completing the following LDC activities? 


A great deal of success Some success 


Little success No success 
a. The reading mini-tasks built into the instructional ladder Oi Or O3 On 
b. The writing mini-tasks built into the instructional ladder O O O O 
c. The final writing task O O O O 


38. In your most recent module, indicate yes or no for each of the following questions: [ note: randomize options, letters don’t appear ] 


Yes No 
a. [have used the LDC rubric to assess my students’ final writing piece. O, Oo 
b. | found the LDC rubric helpful in assessing my students’ final writing piece. O O 
c. 


Using the rubric has helped my students understand the expectations for high quality writing. O O 


39. Please indicate the degree to which you agree or disagree with the following statements. 


Agree Disagree 
Agree Somewhat Somewhat Disagree 


a. l|had sufficient time to prepare to teach modules. Oi Or O3 On 
b. | felt adequately prepared to effectively use modules. O O O O 
c._ Itis difficult to find the time to respond to student writing. O O O O 
d. lam unsure about how best to give productive feedback to student writing. O O O O 
e. Using the LDC modules takes too much time away from covering required curriculum topics. O O O O 
f. It is challenging for me to find content-rich reading materials at my students’ reading level. O O O O 
g. Itis challenging for me to find the time to develop modules. O O O O 


Q40 is about using the LDC modules next year (2013-14). 


40. Please indicate the degree to which you agree or disagree with the following statements: 


Agree Disagree 
Agree Somewhat Somewhat Disagree 


a. | would like to develop modules next year. Oi Oz O3 On 
b. |look forward to teaching modules next year. O O O O 


c. | plan to improve how | teach modules next year. O O O O 


41. Please indicate the degree to which you agree or disagree with the statements below: 
Agree Disagree 
Agree Somewhat Somewhat Disagree 


a. My participation in the LDC initiative is worth the time and effort involved. Oi Oz O3 On 
b. |see the ideas and practices of the LDC initiative gaining traction in my school. O O O O 
c. 1|have noticed an increase in the number of teachers using the LDC modules/tasks in my 

school since last year. O O O O 
d. There are other curricular initiatives or programs in the district that address some of the 

same purposes as LDC. O O O O 
e. The other curricular initiatives or programs in the district create competing priorities 

with the LDC initiative. O O O O 
f. The district has the commitment to sustain the LDC initiative. OQ Oo O Oo 
g. The district has the funding to sustain the implementation of the LDC initiative. O O O O 

42. Have you shared any of your LDC modules with a teacher who is not participating in the LDC initiative? O, Yes Oo No 


SCHOOL LEADERSHIP 


Q43 is about the administrators at your school. 


43. Please indicate the degree to which you agree or disagree with the following statements. 


My school administrators ... 


a. ... have a firm understanding of the LDC framework. 

b. ... have made formative assessment a priority at my school. 

c. ... encouraged me to participate in the LDC initiative. 

d._... provided me with feedback about my instruction of the module(s). 
e. ... provided ongoing support for the implementation of the LDC tools. 
f. _... expressed concerns that teaching modules is taking time away 


from other instructional priorities. 


g. ... have attended professional development about the LDC framework. 


h. ... have communicated how the LDC framework is aligned with other 
school initiatives 


District administrators ... 


i. ... support the LDC framework. 


j. ... encourage my participation in the LDC initiative. 


k. ... provide ongoing support for implementation of the LDC framework. 


|. ... have a firm understanding of the LDC framework. 


m. .. have attended professional development about the LDC framework. 


Agree 


Or 


O00 0 


OO0 0 


Agree 


Somewhat Somewhat 


Oz 


O00 0 


O00 0 


Disagree 


Os 


O00 0 


O00 0 


Disagree 


Ou 


O00 0 


O00 0 


O00 0 


44. Please indicate the degree to which you agree or disagree with the following statements. 


Agree Disagree 
Agree Somewhat Somewhat Disagree 


a. The LDC framework aligns well with my school’s curriculum. O, O, Oz; Oy 
b. The modules help prepare my students for current state assessment(s). 


c. The LDC framework aligns with the Common Core State Standards. 
d. |see the unique value of the LDC framework to address the Common Core State Standards. 


e. The LDC rubric aligns well with my school’s expectations for assessing student writing. 


Q45-46 are about your interactions with your LDC colleagues. 


45. Doyou and your LDC colleagues have regularly scheduled common planning time to discuss LDC? 


O, Yes Oo No 


Agree Disagree 
Agree Somewhat Somewhat Disagree 


46. | would describe my LDC colleagues as collaborative. Oi Oz O3 On 


47. About how often do you have scheduled meetings (as opposed to informal discussions) with your LDC initiative colleagues to discuss 
student work, instructional strategies, or teaching approaches? [skip if Q45=no] 
O, At least once a week 
O, Every other week 
O3 Once a month 
Ox Once per quarter/trimester/semester 
O,; Never 


48. About how often do you have informal discussions (as opposed to scheduled meetings) with your LDC colleagues to discuss student work, 
instructional strategies or teaching approaches? 


O, At least once a week 

Oy, Every other week 

O3 Once amonth 

Oj, Once per quarter/trimester/semester 
O,; Never 


Please indicate the degree to which you agree or disagree with the following statements. 


49. Collaboration with my LDC colleagues helps me... 
Agree Disagree 
Agree Somewhat Somewhat Disagree 


a. ... more effectively use the LDC framework. O, QO; O; Oy 
b. ... better support student learning. O O O O 
c. ... develop LDC modules. O O O O 
d._... teach LDC modules. Oo Oo Oo Oo 
e. ... revise LDC modules. Oo Oo Oo Oo 
f. ... use the LDC framework rubric. Oo Oo Oo Oo 
g. _... use students’ products to inform my instruction. O O O O 
h. ...provide helpful feedback to students about their writing. O O O O 


WORKING WITH EXPERIENCED LDC COLLEAGUES 


50. Are there teachers in your school or district who used LDC modules last year (2011-12)? 


O1 Yes Oo No [skip to 053] 


51. Did you work with a colleague more experienced with LDC this year (2012-2013)? 


Os Yes Op No [skip to 053) 


52. How much did working with a colleague more experienced with LDC help you to develop and teach modules? 


O, a great deal 
QO, a fair amount 
O3 some 

Oj. not much 


Os notatall 


PROFESSIONAL DEVELOPMENT 


53. Have you participated in formal professional development sessions related to LDC during the current school year (2012-13)? 


O1 Yes Oo No [skip to Q61 ] 


54. Which PD providers facilitated the LDC professional development you attended this year (2012-13)? Please CHECK ANY that applies 


C1, State or regional staff 

C1, External partner (e.g., Metametrics, SREB, LDC) 
C1. District or network staff 

C4 School-based staff 


1.1 don’t remember who facilitated the PD this year 


55. How many formal, scheduled LDC professional development sessions have you attended this year (2012-13)? [integer, 1-99] 


56. Please indicate whether you participated in the following types of LDC professional development sessions. 


Participated Did not participate 
a. One-on-one classroom visits O, Oo 
b. Coaching O O 
c. Webinars Oo Oo 
d. Small group meetings O O 
e. School-wide meetings O O 
f. District-wide meetings O O 
g. Cross-district meetings O O 


57. Please indicate whether the LDC professional development sessions you participated in was effective or not effective. 


Not 
Effective Effective 
a. Oo 1 Oo 
b. O O 
C. O O 


58. Please indicate whether the LDC professional development sessions you have participated in contained the following types of content: 


S30 RTT sare aoTD 


Using LDC modules as a way to implement the Common Core State Standards 
Building a teaching task 

Finding appropriate content materials 

Designing modules 

Using the instructional ladder 

Using mini-tasks to address reading and writing skills 
Providing students with feedback on their writing 

Scoring student work with LDC rubric 

Building modules with Module Creator 

Differentiating module instruction to meet student needs 
Implementing modules with special education students 
Implementing modules with ELL students 


. Implementing modules with students who read or write below grade level 


Implementing modules with students with advanced literacy levels 


PD contained 
this content 
Or1 


OCOOOQOQO0O0000000 


PD did not 
contain this content 
Oo 


OCOOQOQO0O0000000 


Q59 asks about LDC professional development that would support your implementation of the LDC initiative. 


59. Please indicate whether you would like more LDC professional development on... 


a) 
b) 
c) 
d) 


e) 


..using LDC modules as a way to implement the Common Core State Standards 
.. building a teaching task. 

.. finding appropriate content materials. 

.. designing modules. 

.. using the instructional ladder. 

.. using mini-tasks to address reading and writing skills. 

.. providing students with feedback on their writing. 

.. scoring student work with the LDC rubric. 

.. building modules with Module Creator. 

.. differentiating module instruction to meet student needs. 

.. implementing modules with ELL students. 

.. implementing modules with special education students. 

.. implementing modules with students who read or write below grade level. 


.. implementing modules with students with advanced literacy levels. 


60. Are you compensated for attending professional development sessions? O, Yes 


Yes 


© 


Oo Oe. OO -O° O OOO: OO 


No 


= 


Ow Ox sO 1D OO: Or oO: OQ 


Oo No 


61. What additional supports and training would help you use the LDC framework? Please use the field below to describe. 


po 


62. Surveys are not perfect. Maybe we missed some things that you think are important about the LDC initiative. Below, we invite you to 


write your assessment and comments about the framework as you have experienced it. 


pT 


63. What is your race/ethnicity? Please CHECK ONE that apply. 
OO, Native American 
Ol, Asian/Pacific Islander 
C1. Black or African American 
C,g Hispanic or Latino 
CO. White or Caucasian 
Oo; Multiracial 
C1, Other (please specify) 


REGULAR CLOSE 
Thank you very much for the time and thought you have put into completing this survey. 


To ensure anonymity, your responses will be combined with those from teachers of numerous schools. 


Your responses will help to inform implementation of the Literacy Design Collaborative. 


DON’T AGREE CLOSE 
We are sorry you have chosen not to participate in the survey. 


Thank you for visiting Research for Action’s and the National Center for Research on Evaluation, Standards, and Student Testing’s survey on the 
Literacy Design Collaborative. 


ERROR MESSAGE IF AN ANSWER IS LEFT BLANK: 
You have not given an answer for a question on this screen. 
Do you want to go back to give an answer or continue with the survey? 
O | want to go back to answer the question. 


O | want to continue without answering the question. 
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ENGLISH LANGUAGE ARTS IN LIFE SCIENCE 


Assessment Directions 


This is a two part assessment. In part 1, you will go over the directions, read the materials, and 
answer questions. In part 2, you will write an essay on the topic provided. 


You will be asked about natural selection in widowbirds and the concept of survival of the fittest. As 
you read the materials please consider the reproductive advantages for long-tailed widowbirds. 


You may take notes directly on the assessment. 
The materials include: 


1. Reading Passage - Survival of the Fittest 
2. Diagram - Natural Selection in Nature 


3. Reading Passage - Widowbirds 


Underlined terms are defined in a glossary. 


Stop at the Stor sign at the end of part 1. 


ENGLISH LANGUAGE ARTS IN LIFE SCIENCE 


E23 1. Reading Passage - Survival of the Fittest 


The passage below is about fitness and how the term is used in discussions of evolution. 


“Survival of the fittest.” This is perhaps the most famous phrase from Darwin’s most famous 
book On the Origin of Species. Unfortunately, this phrase is often misunderstood. We often 
use the word “fitness” to describe someone’s physical strength or health. In discussing 
evolution, fitness is not primarily a measure of strength or health. It has everything to do with 
an organism’s reproductive success. Specifically, how many offspring it has in a lifetime, 
compared to other individuals in the population. 


There are three important elements to an organism’s fitness: 


1. An individual's fitness is measured relative to the fitness of individuals with different 
genotypes or phenotypes in the population. 


2. Fitness depends on the specific environment in which the organism lives. 


3. Fitness is measured by comparing an organism’s reproductive success with the 
reproductive success of other organisms in the population. 


To illustrate the concept of fitness, consider the following example. 


Fly A carries a trait that allows it to survive a long time without food. Fly B carries a trait that 
allows it to survive only a short time without food. If there is a long period of time when 
there is no food, Fly A will be more likely to survive and have a chance to reproduce. Thus, 
Fly A exhibits greater fitness in environments where food shortages occur more frequently. 


continue reading p> 


allele - different versions of the offspring - progeny or descendants of a 
same gene; for instance, allele “A” person, animal, or plant 
produces a tall plant while allele “a” 


produces a short plant phenotype - appearance or observable 


characteristics of an organism (e.g., a 
long tail) resulting from the interaction of 
its genetic makeup with the environment 


genotype - the genes that an 

organism possesses for a 

particular trait; also, the entire trait - a distinguishing characteristic 

genetic makeup of an organism (e.g., eye color) passed from parent to 
offspring 


ENGLISH LANGUAGE ARTS IN LIFE SCIENCE 


To further illustrate this point, consider some additional information about these flies: 


Fly A: produces 40 eggs per week and can survive longer without food. 
Fly B: produces 60 eggs per week and can't survive very long without food. 


Environment 1: frequent food shortages. 
Fly A: has an expected lifespan of five weeks. Lifetime egg output = 200 
Fly B: has an expected lifespan of two weeks. Lifetime egg output = 120 


Environment 2: no food shortages. 
Fly A: has an expected lifespan of five weeks. Lifetime egg output = 200 
Fly B: has an expected lifespan of five weeks. Lifetime egg output = 300 


Whichever fly has greater fitness over a lifetime depends on the environment and requires a 
calculation of reproductive output (not just a measure of survival). For this reason, “survival 
of the fittest” can be a misleading phrase. It can lead some to believe that survival has to do 
with health and strength. It becomes clearer if we understand that reproductive success 
determines whether an organism will ultimately pass on its genes to the next generation. 
Individuals with alleles that increase their fitness will be more likely to survive and 
reproduce. This will lead to an increase in those alleles in the population and a decrease in 
alleles that don’t increase fitness. 


Adapted from Phelan, J. (2010). What is Life? A Guide to Biology, NY: W.H. Freeman and Company 
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% Reading Comprehension Questions 


In this part of the assessment, you will answer questions about the content of the passage you 
read. You may look back at the materials to help you answer the questions. 


Multiple Choice 


Choose the best answer for each question. Fill in the bubble next to the answer you choose. 


1. Which of the following best describes the physical characteristic expressed by 
genes? 
@) phenotype 
genotype 
() fitness 


(©) reproductive success 


2. Which of the following best describes the ability of an individual to survive and 
reproduce in its specific environment? 


@) genotype 


allele 
©) fitness 


©) reproductive output 


3. Which of the following provides the best summary of the passage? 


@) Phenotypes and genotypes influence an individual’s fitness. 


Fitness for an individual is largely determined by reproductive success and not physical 
fitness. 


(©) Darwin’s famous phrase should be changed so that it doesn’t confuse readers. 


Trait variation in fruit flies can enhance reproductive success. 
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4. What is the main purpose of the passage? 


@) compare and contrast different points of view 
discuss results of a study 
(©) clarify a misconception or misunderstanding 


(©) describe a scientific theory 


Short Answer 


5. In 1-2 sentences, explain why Fly A is more fit than F/y B in environments in which 
there are frequent food shortages. 
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Ee 2. Diagram - Natural Selection in Nature 

The diagram below illustrates the conditions that must be met for natural selection to occur. Natural 
selection can be thought of as getting rid of traits that lead to poor reproductive success. If you are 
a slower-running rabbit, you are more likely to be eaten by a fox. And so the next generation of 
rabbits will have fewer slow rabbits. 


NATURAL SELECTION IN NATURE 


@ VARIATION FOR A TRAIT 
Running speed in rabbits can vary from one individual to the next. 


! 
ge Speed =———> 
nd 


@® HERITABILITY 


The trait of running speed is passed on from parents to their offspring. 


@ DIFFERENTIAL REPRODUCTIVE SUCCESS 
In a population, rabbits with slower running speeds are eaten by the 
fox and their traits are not passed on to the next generation. 


Source: Phelan, J. (2010). What is Life? A Guide to Biology, NY: W.H. Freeman and Company. 
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% Reading Comprehension Questions 


In this part of the assessment, you will answer questions about the content of the diagram. You may 
look back at the materials to help you answer the questions. 


Multiple Choice 
Choose the best answer for each question. Fill in the bubble next to the answer you choose. 


1. Which of the following best describes the process where traits become more or less 
common in a population? 


(@) fitness 


reproductive success 
() natural selection 


(©) heritability 


2. What is the main idea of the diagram? 


@) Trait variation helps determine survival because species with advantageous traits survive to 
reproduce and pass those traits to offspring. 


Characteristics acquired after birth explain how certain members of a population survive over 
others. 


Natural selection and reproduction are ways to understand similarities between species. 


© © 


Slower running rabbits tend to get eaten by foxes at a higher rate than faster rabbits. 
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Short Answer 


3. If all the rabbits in a population have the same running speed, would the process of 
natural selection as shown in the diagram still occur? Explain your answer in 1-2 
sentences. 


ENGLISH LANGUAGE ARTS IN LIFE SCIENCE 


&3 3. Reading Passage - Widowbirds 


The passage below provides a short description of a study investigating female mate choice in a 
group of birds called widowbirds. 


Adult male widowbirds are black and have tails that can be as long as 50 cm. That’s up to 
three times as long as their bodies (see photograph below). 


Female widowbirds are brown, with much smaller tails. Female widowbirds make nests in 
the males’ territories and raise two to three young without any assistance from the males. 


5 Researchers decided to conduct an experiment to study mate selection in several groups of 
male widowbirds. One group of widowbirds had their tails artificially lengthened by gluing 
additional tail feathers onto their tails. Another group had their tails shortened. This left one 
group of birds with longer tails and one group of birds with 
shorter tails. 


10 At the start of the experiment, researchers counted the number 
of nests (those with eggs or young birds) on the territory of each 
male. After one month, the researchers compared the number of 
nests in each territory. The group of males (long or short tail) 
with the highest number of nests would determine which group 

15 was more successful in reproducing. 


Results of the study showed that the reproductive success of the 
males with longer tails was significantly higher than males with 
shorter tails, although the longer-tailed males were more 
susceptible to predation and some had difficulty flying. 


artificially - not naturally susceptible - likely to be harmed 
by something 


predation - when one species attacks 
and/or kills another species territory - a defined area of land 
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20 The following illustrates the reproductive success over the average lifespan of the different 
birds: 


e Long-tailed widowbirds produce an average of five offspring each year. Because they 
only live an average of two years (because they are more susceptible to predation and 
can't fly well), they have an average expected lifetime output of ten offspring. 


25 e Short-tailed widowbirds produce an average of one offspring per year (because fewer 
females select them) and live an average of six years. Therefore, their average 
expected lifetime output is six offspring. 


So we see that long-tailed widowbirds have greater fitness, even with a shorter life. Hence, 
reproduction is more important than survival when it comes to the concept of fitness. 


Adapted from 
Andersson, M. (1982). Female choice selects for extreme tail length in a widowbird. Nature, 299, 818-820. 


Linn, Currie. (Photographer). (2010). Long tailed widow bird in natural environment, showing off long tail [Photograph]. Retrieved from 
http://www.shutterstock.com/pic-23789029/stock-photo-long-tailed-widow-bird-in-natural-environment-showing-off-long-tail.html 
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% Reading Comprehension Questions 


In this part of the assessment, you will answer questions about the content of the passage you 
read. You may look back at the materials to help you answer the questions. 


Multiple Choice 


Choose the best answer for each question. Fill in the bubble next to the answer you choose. 
1. What does the author mean by “more susceptible to predation” (lines 18-19)? 


() The longer-tailed birds are at greater risk of being attacked by predators. 
Predators prefer the short-tailed to the long-tailed birds. 
©) The longer-tailed birds are better able to escape from predators. 


©) The widowbirds have no known predators. 


2. According to the passage, what could the researchers conclude based on the results 
of the study? 
(A) Widowbirds with short tails have a greater chance of survival. 
Trait variation in species can impact flying. 
©) Reproductive success is related to tail length in male widowbirds. 


(©) Female widowbirds have much shorter tails than male widowbirds. 


3. What is the main purpose of the passage? 


@) discuss the mating habits of the widowbirds 
compare and contrast the tail length of male and female widowbirds 
(©) describe the process of artificially lengthening widowbirds' tails for an experimental study 


©) report findings from an experiment on the mating success of widowbirds with different tail 
lengths 
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4. What evidence is used in the passage to show that males with longer tails are more 
successful in reproduction? 
@) A smaller number of males with longer tails were seen being attacked by predators. 
There were more eggs in the nests of longer-tailed males. 
(©) The shorter-tailed males were seen flying without difficulty. 


(©) Female widowbirds were more frequently seen with longer tailed males. 


Short Answer 


5. How would the result of the study change if short-tailed widowbirds lived an 
average of 12 years? 


12 


End of Part 1 
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Writing Task 


Using evidence from the materials provided and other information you know about “survival of the 
fittest”, write a persuasive essay in which you argue against the following statement: 


Results presented in the reading passage “Widowbirds” clearly show that the widowbirds 
with longer tails have lower fitness because they are more susceptible to predation. After all, 
Darwin explained that evolution is all about survival of the fittest. 


In your persuasive essay, be sure to explain how birds with reduced life spans may actually be 
considered more “fit” from an evolutionary perspective than those with longer life spans. 


In your formal essay, be sure to: 


e briefly describe “survival of the fittest” in your own words. 


e support your argument with examples from (a) the passages and diagram provided, and (b) 
general concepts and specific facts you already know about “survival of the fittest” and 
natural selection. 


Your essay will be scored on how well you: 


e demonstrate an understanding of “survival of the fittest.” 

e state and support your claims with evidence. 

e present your essay in a logical and well-organized manner. 
e use the materials to support your argument. 


e use proper punctuation, spelling, and grammar. 
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Assessment Directions 


This is a two part assessment. In part 1, you will go over the directions, read the materials, and 
answer questions. In part 2, you will write an essay on the topic provided. 


You will be asked about the period called Reconstruction that followed the Civil War. As you read 
the materials, please consider the how each document speaks to the challenges faced by former 
slaves during Reconstruction. 

You may take notes directly on the assessment. 


The materials include: 


1. Timeline - Key Events during Reconstruction 

2. Reading Passage - Frederick Douglass 

3. Reading Passage - South Carolina's Black Codes 
4. Reading Passage - KKK 


Underlined terms are defined in a glossary. 


Stop at the sroP sign at the end of part 1. 
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yc eq 1. Timeline - Key Events during Reconstruction 


Reconstruction refers to the historical period immediately following the Civil War (1865-1877), when 
the nation faced the dual challenges of rebuilding national unity and addressing the needs of four 
million freed slaves. The North won the war and controlled Congress during Reconstruction. Most 
Northerners were Republicans and, in general, supported the rights of freed slaves in the South. 
During Reconstruction, Republicans in Congress used the U.S. Army to protect freed slaves against 
violence in the South. When these troops left the South in 1877, Reconstruction effectively ended. 


1865: The Civil War ends, and Republican President Abraham Lincoln is assassinated. 
13'" Amendment outlaws slavery. 


Many Southern states create Black Codes to limit rights of former slaves. 


1866: The Civil Rights Act of 1866 allows African Americans to own property and to be 
treated equally in court. 


The Ku Klux Klan is founded. 


1867: Radical (extreme) Republicans take over the United States government and pass many 
laws to support the rights of former slaves. 


1868: The 14" Amendment grants citizenship to African Americans. 


First African American elected to United States Congress. 
1870: The 15" Amendment grants African Americans the right to vote. 


1871: Congress passes the Ku Klux Klan Act of 1871 in response to reports of widespread 
violence against African Americans in the South. The act gave the U.S. Army the power 
to enforce the rights of African Americans. 


1877: Reconstruction ends. The government pulls all remaining Northern troops out of the 
Southern states. 


|e O 
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% Reading Comprehension Questions 


In this part of the assessment, you will answer questions about the timeline you read. You may look 
back at the materials to help you answer the questions. 


Multiple Choice 


Choose the best answer for each question. Fill in the bubble next to the answer you choose. 


1. Based on the timeline, what was the major tension between the North and the South 
during Reconstruction? 


@) Northern Republicans and Southern Democrats disagreed about whether African Americans 
should be elected to Congress. 


Southern Democrats supported the Ku Klux Klan, but Northern Democrats wanted it 
abolished. 


©) Southern Democrats wanted to pass constitutional amendments, but the North opposed them. 


©) Northern Republicans wanted to establish rights for freed slaves, while the South resisted. 


2. During Reconstruction, constitutional amendments were passed that guaranteed the 
following rights to African Americans, EXCEPT the right to: 
@) own property. 
citizenship. 


() be free. 
©) 


vote. 
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3. What does the Ku Klux Klan Act of 1871 tell you about the United States during 
Reconstruction? 
@) The South used military force against the North in the years following the Civil War. 
The South was committed to protecting the rights of African Americans. 
(©) The South wanted the North’s help to enforce the policies of Reconstruction. 


©) The North and the South still did not see eye to eye after the Civil War. 


Short Answer 


4. In 1-2 sentences, describe how you think life changed for the African Americans in 
the South after Northern troops withdrew in 1877. 
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&3 2. Reading Passage - Frederick Douglass 


The following excerpt is from a speech delivered by Frederick Douglass in August 1880 at a 
meeting in Elmira, NY. Douglass escaped from slavery to become a leading advocate of 
abolitionism before the Civil War. In the years following the Civil War, he was a leading proponent 
of African American civil rights. 


How stands the case with the recently emancipated millions of Negro people in our own 
country? By law, by the Constitution of the United States, slavery does not exist in our 
country. By law and the Constitution, the Negro is a man and citizen, and has all the rights 
and liberties guaranteed to any other type of the human family, living in the United States. 


5 But today, in most of the Southern states, the Fourteenth and Fifteenth Amendments are 
virtually nullified. The citizenship granted in the Fourteenth Amendment is practically a 
mockery, and the right to vote, provided in the Fifteenth Amendment, is literally stamped 
out. The old master class is today triumphant. 


The very manner of their Emancipation invited the bitterest hostility of race and class to the 

10 freedmen. They were hated because they had been slaves, hated because they were now free, 
and hated because of those who had freed them. Nothing was to have been expected other 
than what has happened. The old master class would naturally employ every power and 
means in their reach to make the great measure of Emancipation unsuccessful. When the 
Hebrews were emancipated, they were told to take spoil from the Egyptians. When the serfs 

15 of Russia were emancipated, they were given three acres of ground upon which they could 
live and make a living. But not so when our slaves were emancipated. They were sent away 
empty-handed, without money, without friends and without a foot of land upon which to 
stand. Old and young, sick and well, were turned loose to the open sky, naked to their 
enemies. 


Adapted from Life and Times of Frederick Douglass (p. 610), by F. Douglass, 1882, Hartford, Connecticut: Park Publishing. 


nullified — of no use or value virtually — for the most part 


spoil — stolen goods 
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% Reading Comprehension Questions 


In this part of the assessment, you will answer questions about the content of the passage you 
read. You may look back at the materials to help you answer the questions. 


Multiple Choice 


Choose the best answer for each question. Fill in the bubble next to the answer you choose. 
1. What did Douglass say about the Fifteenth Amendment? 


@) It granted former slaves the right to vote. 
It granted former slaves the right to citizenship. 
©) It was ignored by Southern states by 1880. 


(©) It made the master class triumphant. 


2. According to Douglass, how does the situation of the newly emancipated slaves in America 
compare to that of the Russian serfs or the Hebrews? (lines 13-16) 
@) The Russian serfs and Hebrews were sent away after emancipation. 
The Russian serfs and Hebrews were given land after emancipation. 
©) The African American slaves were given land after emancipation. 


(©) The African American slaves were given nothing after emancipation. 


3. To whom is Douglass referring when he talks of the “old master class”? 


@) all Southerners 
former slave owners 
(©) US. Citizens 


©) Republicans 
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Short Answer 


4. Ina few sentences, summarize the main points of Douglass’ speech. 


ENGLISH LANGUAGE ARTS IN US HISTORY 


E23 3. Reading Passage - Southern Carolina's Black Codes 


In the years immediately following the Civil War, many Southern states passed "Black Codes,” or 
laws that tried to control former slaves. The excerpt below is from South Carolina's Black Code, 
passed in December 1865. 


An Act to establish and regulate the Domestic Relations of Persons of Color 


Contracts for service: All persons of color who make contracts for service or labor, shall be 
known as servants, and those with whom they contract, shall be known as masters. 


Regulations of labor on farms: The hours of labor, except on Sunday, shall be from sunrise to 
sunset, with a break for breakfast and dinner. Servants shall rise at the dawn in the morning, 
do the usual and needful work about the premises and begin the farm work by sunrise. 


Mechanics, Artisans and Shop-Keepers: No person of color shall pursue or practice the art, 
trade or business of an artisan, mechanic or shop-keeper, or any other employment (besides 
that of farming, or that of a servant), until he shall have obtained a license from the Judge of 
the District Court, which shall be good for one year only. 


Vagrancy and Idleness: All of the following people shall be considered vagrants: 


e People who do not have some fixed and known place of abode, and some lawful and 
respectable employment; 

e People who lead idle or disorderly lives; 

e People who are able to work and do not work; 

e People who perform publicly or privately, for fee or reward, any tragedy, comedy, play, or 
other similar entertainment; 

e People who for private gain, give any concert or musical entertainment, of any description; 
fortune-tellers; beggars; 


On conviction, the defendant shall be imprisoned and/or sentenced to hard labor. 


The defendant, if sentenced to hard labor, may be hired to any owner of a farm for the term of hard 
labor to which he was sentenced. 


Adapted from Acts of the General Assembly of the State of South Carolina Passed at the Sessions of 1864-65 (Columbia: 1865), pp. 291- 
304 


artisan — a worker in a skilled trade vagrant — a person without a settled 
home or regular work who wanders 


abode — house or home from place to place; beggar 
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% Reading Comprehension Questions 


In this part of the assessment, you will answer questions about the content of the passage you 
read. You may look back at the materials to help you answer the questions. 


Multiple Choice 


Choose the best answer for each question. Fill in the bubble next to the answer you choose. 


1. According to the South Carolina Black Codes, freed slaves could only work as 
which of the following: 


@) public performer 


business owner 
©) artisan 


(©) farm laborer 


2. How did the Black Codes help the owners of plantations? 


(@) The law gave plantation owners licenses to pursue other forms of employment. 
The law forced freed slaves to work for their former owners on plantations. 
©) The law made sure that farm labor received breaks for breakfast and dinner. 


(©) The law made sure that no vagrants were allowed on their land. 


3. The punishment for someone arrested for vagrancy included which of the following: 


@) They might be sent out of the state. 
They might have to obtain a license from a district judge. 
(©) They might be sent to work on a farm. 


©) They might be prevented from future work. 
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Short Answer 


4. Choose ONE regulation that supports the argument that the authors of the Black Codes 
wished to reestablish the conditions of slavery. Explain your choice. 
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Reading Passage - KKK 


In 1871, Congress held hearings to gather information about widespread violence against African 
Americans in the South. These hearings resulted in the passage of the Ku Klux Klan Act of 1871, 
which allowed the President to use federal troops to suppress racial violence. The transcript below 
comes from the congressional hearings. 


[Harriet Hernandez and her husband were whipped by the Ku Klux Klan in South Carolina. 
Local whites apparently became outraged after Harriet’s husband rented some land and she 
subsequently withdrew from domestic labor. Here she recounts being compelled to ‘lie out’ 
in the woods at night to avoid further violence]: 


Q: 


Ae 


Had [your husband] been afraid for any length of time? 


He has been afraid ever since last October. He has been lying out. He has not slept in the 
house ten nights since October. 


: Is this situation similar to that of other colored people down there to any extent? 
: That is the way they all have to live —men and women both. 

: What were they afraid of? 

: Of being killed or whipped to death. 

: What has made them afraid? 


: Because if the men voted [Republican], they took the spite out on the women when they 


could get at them. 


: How many colored people have been whipped in that neighborhood? 


: It is all of them, mighty near. I could not name them all.... They have no satisfaction to 


live like humans, no how. 


Source: [KKK Hearings, SC: 586] Joint Select Committee to Inquire into the Condition of Affairs in the Late Insurrectionary States [South 
Carolina], Washington, D. C., 1872 
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Reading Comprehension Questions 


In this part of the assessment, you will answer questions about the content of the passage you 
read. You may look back at the materials to help you answer the questions. 


Multiple Choice 


Choose the best answer for each question. Fill in the bubble next to the answer you choose. 


1. What word best describes how Harriet and her husband felt in 1871? 


(a) sad 
hopeful 
©) confused 


(©) afraid 


Which of the following actions by African Americans angered local whites in South 
Carolina in 1871? 


() sleeping at home 
lying in the woods 
©) voting Republican 


(©) working as domestic labor 


Was Harriet’s experience typical for African Americans in South Carolina in 1871? 


@) yes, according to this testimony 
no, according to this testimony 
©) maybe, according to this testimony 


©) this testimony does not say 
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Short Answer 


4. a. Based on this testimony, did African Americans feel free to exercise their right to 
vote in 1871? 


b. Explain your answer using evidence from the document. 
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End of Part 1 
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Writing Task 


Even though the Thirteenth Amendment of 1865 officially ended slavery in the United States, former 
slaves faced many challenges in the following years. Using the documents as evidence, write a 
formal essay arguing that African Americans were not actually free in the years immediately 
following the Civil War. 


Your essay will be scored on how well you: 


e demonstrate an understanding of Reconstruction. 

e state and support your claims with evidence. 

e present your essay in a logical and well-organized manner. 
e use the materials to support your argument. 


e use proper punctuation, spelling, and grammar. 
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Exhibit A6: ILA Writing and Short Answer Rubrics 


CRESST ILA Scoring Rubric for Grade 8 Writing 


Overview 


The five scoring dimensions are: content understanding, rhetorical structure/quality, reference to text, and 
use of grammar/conventions. There may be some overlap between some of these dimensions, which is fine. 
When scoring, keep in mind we are looking at on overall indicator of proficiency on each of these 
dimensions. 


Dimension Name Description 


A Content understanding This is a measure of overall how well the student has 
demonstrated that they understand the materials and the 
topic in their essay. 


B Rhetorical structure/quality Argument: Measures how well the elements of an argument 
are described in the response. For 8" grade writing, 
arguments should establish a claim, distinguish claims from 
alternate or opposing claims, and support claims with 
relevant evidence and logical reasons. 


C Organization This evaluates the focus, logical progression of ideas, and 
structure demonstrated by the student's writing. 
D Reference/support with text This is a measure of how well statements in the essay are 


supported by references to text details. A text detail is a 
quotation, paraphrase, or any other reference to information 
and ideas in the texts provided. 


E Grammar and Conventions Evaluates the command of standard English conventions 
demonstrated by the response: proper English usage and 
control of grammar, formal tone, correct paragraph and 
sentence structure. 


Student ID: 


Score point 


Argument Rubric 


Score 


Content 
understanding 


The response demonstrates well-developed and thorough understanding 
of the topic. 


O 


The response demonstrates solid understanding of the topic. 


The response demonstrates some understanding of the topic. 


The response demonstrates little or no understanding of the topic. 


Rhetorical 
structure/quality 


Important elements of the argument are clearly and thoroughly described 
and articulated and the response is aligned to the writing prompt. 


Elements of the argument are clearly described and the response is 
aligned to the writing prompt. 


There is an attempt to describe some elements of the argument. 


Elements of the argument are not described, or the descriptions are 
unclear. 


Organization 


The essay is well-organized. 


The essay is appropriately organized. 


The essay is somewhat organized. 


The essay has little or no appropriate organization. 


Reference 
support with 
text 


The response uses detailed and well chosen references to the text to 
thoroughly support the argument. 


The response uses accurate and detailed references to the text to provide 
solid support for the argument. 


The response uses some accurate and detailed references to the text to 
provide support for the argument. 


The response uses little or no accurate and detailed references to the text 
to provide support for the argument. 


Grammar and 
Conventions 


The response demonstrates a well-developed command of standard 
English conventions. 


The response demonstrates an appropriate command of standard English 
conventions. 


The response demonstrates some command of standard English 
conventions. 


The response demonstrates little or no command of standard English 
conventions. 


—/ NO) O/B] —)/ NO) |) B/N} |B) =) Nh) ) BR) —/ Nh) O) B 
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Evolution Short Answer Rubric 


O O 
Question ID Q# | Question Stem & Sample Scoring Rubric 
Answer 
MSC_SF_ 160 5 In 1-2 sentences, explain why BL = Blank 
Fly A is more fit than Fly B in 0 = The student gives either an 
environments in which there are | incomplete or incorrect 
frequent food shortages? explanation. 
1 = The student provides a 
Answer: In environments with correct explanation for why Fly 
frequent food shortages, Fly A _| A is more fit in environments 
is more fit than Fly B because it | with frequent food shortages: it 
has a greater lifetime egg output | (Fly A) has more eggs in its 
under those conditions. Fitness | lifetime (greater reproductive 
reflects relative reproductive success). Because of the stem, it 
success, So Fly Ais more fitin | is not necessary to specify that 
those circumstances. this is about Fly A as compared 
to Fly B, or that it is only true in 
an environment with frequent 
food shortages. 
MSC_NSN_161 | 3 If all the rabbits ina population | BL = Blank 


have the same running speed, 
would the process of natural 
selection as shown in the 
diagram still occur? Explain 
your answer in 1-2 sentences. 


Answer: No. Natural selection 
would not occur as shown in the 
diagram. If the rabbits all had 
the same running speed, they 
would be equally vulnerable to 
predation. It would not matter 
which rabbits are eaten by foxes 
because all rabbits have the 
same running speed and without 
variation for a trait, natural 
selection cannot occur. 
Regardless which rabbits escape 
the foxes and reproduce, their 
offspring would always inherit 
the same running speed. 


0 = Incorrect response. 

1 = The student gives the 
correct response (no*) but 
insufficient reasoning. 


2 = The student responds 

correctly and supports their 

response with reasoning. The 

response should mention (or 

clearly imply) reproduction: 

¢ No*, because the rabbits that 
(live and) reproduce would 
be the same as the rabbits that 
are eaten. 

¢ No*, because the offspring 
would all inherit the same 
running speed. 


* Any accurate description of 
the lack of natural selection is 
the same as “No”, e.g. “the next 
generation of rabbits would be 
the same because...’, or 

“, therefore the offspring 
wouldn’t be faster”. 


Reconstruction Short Answer Rubric 


Reconstruction MS, v3 


main points of Douglass’ speech? 


Answer: Douglass’ main point is 
that although by law slavery has 
been outlawed, emancipation has 
been unsuccessful because the 
laws (14"" and 15"" amendment) 
have largely been ignored. The 
“Freedmen” faced so much 
hostility because the old master 
class did not want emancipation to 
be successful. Unlike other freed 
people who were emancipated, 
African American freedmen 
received no resources to transition 
out of slavery (e.g., land, money). 


Question ID Q#_ | Question Stem & Sample Answer_| Scoring Rubric 
MSO_RC 200 /|4 In 1-2 sentences, describe how you | BL = Blank 
think life changed for the African | 0 = The student either gives 
Americans in the South after an incomplete or incorrect 
Northern troops withdrew in 1877. | response, e.g. the response 
does not describe a change. 
Answer: Without the presence of | | = The student responds 
Northern troops to protect their correctly. A correct response 
rights, African Americans in the could be a general statement 
South are denied some of their about having fewer rights 
rights and freedoms and are after reconstruction or at 
abused by groups like the KKK. least one example of a 
challenge they would face 
(e.g. discrimination, 
violence) . 
MSO_FD_197 4 In a few sentences, summarize the | BL = Blank 


0 = The student either gives 
an incomplete or incorrect 
response, e.g. the response 
focuses on one main idea 

1 = The student responds 
correctly with at least two 
main ideas from Douglass’ 
speech. 


Exhibit A7: 


LDC CRESST Evolution Assessment: Short Teacher Survey 


Please answer the following questions about your instruction in the 2012-13 school year. 
Answers should reflect your instruction for the classroom in which you are administering the 
enclosed assessments. 


1. 


2. 


3. 


What date(s) were the enclosed assessments administered? 


How much total time did students spending taking the assessments? minutes 
How many LDC modules have you taught in the 2012-13 school year? 


What topics did the LDC modules you taught cover? 


a. 
b. 


o 


am 


Please indicate the degree of emphasis you placed on Evolution as a topic during 
your 2012-13 instruction in this class, by circling the best response below. 


No emphasis Slight emphasis Moderate emphasis Sustained emphasis 


If there were any problems with the administration of the assessments, or you have 
any other comments, please use the space below. 


Exhibit A8: 


LDC CRESST Reconstruction Assessment: Short Teacher Survey 


Please answer the following questions about your instruction in the 2012-13 school year. 
Answers should reflect your instruction for the classroom in which you are administering the 
enclosed assessments. 


1. What date(s) were the enclosed assessments administered? 


2. How much total time did students spending taking the assessments? minutes 
3. How many LDC modules have you taught in the 2012-13 school year? 


4. What topics did the LDC modules you taught cover? 


a. 
b. 


o 


a 


5. Please indicate the degree of emphasis you placed on Reconstruction as a topic 
during your 2012-13 instruction in this class, by circling the best response below. 


No emphasis Slight emphasis Moderate emphasis Sustained emphasis 


6. If there were any problems with the administration of the assessments, or you have 
any other comments, please use the space below. 
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Developing an Assignment Measure of Common Core State Standards Literacy Practice 


The National Center for Research on Evaluation, Standards and Student Testing (CRESST) is 
developing evaluation tools to support the transition to the Common Core State Standards (CCSS). 
Generously supported by the Bill and Melinda Gates Foundation, the CRESST tools include measures of 
both teacher practice and student learning. The Assignment Measure tool presented here examines how 
well 8th grade science and social studies teachers were able to incorporate selected CCSS standards in 
English Language Arts into classroom curriculum and instruction. The tool specifically measures how 
teachers implemented the Literacy Design Collaborative (LDC) framework, an intervention designed to 
help teachers create integrated literacy and content instruction around core writing tasks. 

Assignment and artifact collection has been identified as an efficient and economical means of 
measuring the quality of classroom instruction (e.g., Clare & Aschbacher, 2001; Newmann, Bryk, & 
Nagaoka, 2002; Storms, Riazantseva, & Gentile, 2000; Matsumura, Slater, Wolf, Crosson, Levison, 
Peterson, Resnick, Junker, 2006; Martinez, Borko, Stetcher, Luskin, & Kloser, 2012). Classroom 
assignments shed light on such variables as teacher clarity, cognitive rigor of instruction, and in this case, 
degree of LDC fidelity of implementation. Careful assessment of assignments can potentially capture 
instructional quality with a degree of accuracy that approaches actual observation. In addition to shedding 
light on the nuances of instructional practice, artifact collection and analysis can potentially be leveraged 
for professional development purposes, to model and improve effective practices. The CRESST 
Assignment Measure was designed with both of these uses—instructional evaluation and professional 
development—in mind. Furthermore, the benefit of the Assignment Measure is its transferability: with 
only minor modifications, the measure can be used to assess to any instruction that involves reading texts 
and writing about them. 

In contrast to previous artifact studies that examine discrete assessment practices or assignments 
(cf. Martinez, Borko, Stetcher, Luskin, & Kloser, 2012), the CRESST Assignment Measure was designed 
to capture instruction that occurred within the framework of an LDC “Module,” a unit of instruction 
lasting 2-4 weeks that culminates in a summative writing task. In building an LDC Module, teachers 
begin with fill-in-the-blank template tasks—or extended writing prompts—that are designed to help them 
incorporate the literacy demands specified in the Common Core State Standards into units of instruction 
in literature, history/social studies, or science. Teachers design instructional activities using the LDC 
framework that is comprised of four “skill clusters:” (1) Preparing for the Task, (2) Reading Process, (3) 
Transition to Writing, and (4) Writing Process. Each LDC Module includes an instructional ladder of 
“mini-tasks” that build the requisite skills in both reading and writing to complete the final writing task. 


The final product—instructional ladder plus template task—is referred to as an LDC module. 


LDC has also designed Module Creator, an online platform that walks teachers through the four 
steps of building a module: What task? What skills? What instruction? What results? This platform is 
replete with pull-down menus and default settings that teachers are invited to augment and/or revise, 
depending on their instructional needs. For example, in the Reading Process skills cluster, teachers are 
invited to add additional skills beyond the ones suggested by LDC: text selection, active reading, essential 
vocabulary, academic integrity, note-taking. Likewise, if they choose to teach the skill of “active 
reading,” they have the option of adding additional instruction or “mini-tasks” beyond the default option 
on Module Creator, which simply has students “brainstorm ways to figure out any author’s intent” and 
“share and discuss their answers for each text.” 

The CRESST Assignment Measure was designed to assess both the clarity and quality of the 
writing task and the appropriateness and relevance of the activities in the instructional ladder. Despite the 
fill-in-the-blank format of the final writing task, the process of building an LDC module leaves 
tremendous room for variation. Teachers are responsible for choosing the topic, selecting appropriate 
texts, and designing daily instruction around all elements of literacy, from reading comprehension, 
paragraphing, to all aspects of writing style and mechanics. This potentially wide range in teacher 
implementation has important implications for the design of fidelity of implementation measures. The 
breadth of instruction contained in a given LDC Module poses logistical challenges for artifact collection, 
which will be discussed below. 

In the report below, we describe the CRESST Assignment Measure and report findings from a 
reliability study. In particular, we examine variation and sources of error in ratings, attending to what 
these results might suggest about teacher implementation of LDC. Lastly, using both the quantitative 
results from the reliability study, as well as qualitative data from interviews and surveys, we make 


suggestions for how the Assignment Measure might be used in the future. 


CRESST Assignment Measure 

CRESST’s LDC Assignment Measure was designed to capture the specific demands of 
integrating content and literacy, as is required in subject area classrooms implementing LDC. Over the 
past year, CRESST has been involved in revising, piloting, and validating the LDC Assignment Measure, 
based on findings from last year’s pilot. Our original measure consisted of eight dimensions, each 
targeting a discrete component of the module. Each dimension was scored on a four-point scale. Our 
current measure has nine dimensions, each scored on a 5-point scale. The revised version of the 
assignment measure focuses on rigor and content literacy, while also drawing more explicitly on the 
criteria for “what makes a great teaching task” and “what makes a great module” that are discussed in The 


1.0 Guidebook to LDC (See Appendix A for complete rubric). 
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The final measure includes only those areas for which we are able to collect sufficient evidence of 
classroom practice. The challenge with each of these potential domains of LDC implementation is the 
availability of information to support their assessment. The final rubric assumes that scorers have the 
following materials before them: 1) a completed template task (often printed from Module Creator); 2) 
one sample of supplemental instructional materials in reading instruction and one sample of instructional 
materials in writing instruction (e.g., graphic organizers, worksheets, lesson plans) that speak to the 
specificity of instruction; 3) three samples of student work on the template task, marked high, medium, 
and low; and 4) a short cover sheet where teachers indicate how long they spent teaching the modules and 
state their goals for content, reading, writing. Below, we make suggestions for additional classroom 
artifacts that might be collected in future iterations of the Assignment Measure to assist raters in making 
inferences about classroom instruction. 

Each dimension was scored on a five-point scale for scoring to reduce any clustering or bias 
towards the mean. A score of | indicates that a dimension is not present or realized in the assignment 
measure artifacts; a score of 3 suggests that a dimension is moderately present or realized; and a score of 
5 indicates that the dimension is fully realized. The first five dimensions in the revised measure address 
the question, “what makes a great teaching task?” These dimensions center on the teaching task rather 
than the module in its entirety. This approach aligns with the steps suggested for building an effective 
module, as outlined in The 1.0 Guidebook to LDC (cf. Crawford, Galiatsos, Lewis, & Ottesen, 2011): 

1) Effective Writing Task considers the degree to which the teaching task makes effective use of 

the template task’s writing mode (i.e., argumentation or explanation); requires sustained writing 

and effective use of ideas and evidence to substantiate claims; and is feasible for most students to 
complete (1.e., appropriate for the grade-level and subject matter). Sources of information for 
evaluating this dimension include the teaching task template options, the teaching task, and 


summary information sections of Module Creator. 


2) Alignment to the CCSS and Local and State Literacy and Content Standards focuses on the 
extent to which teaching task addresses content essential to the discipline, as well as reading 
comprehension and writing standards informed by local and state standards. Evidence for 
assessing this dimension can be found in the CCSS and state standards provided in Module 


Creator, as well as the literacy and content standards that teachers include in the module. 


3) Text Alignment is the degree to which the assigned texts address teaching task content. The 


main sources of information for evaluating this and the following two dimensions are the reading 


texts themselves, as well as the task, resources, and links listed in the Module Creator Handout or 


referenced in the student work. 


4) Text Appropriateness is the degree to which the teaching task includes reading texts that are 
accessible to most students (1.e., appropriate for the grade-level and subject matter). Rater 
materials included sample readings that represented 8th grade Lexile levels. Raters were 


encouraged to refer to these exemplar texts when assessing text appropriateness. 


5) Text Rigor is the degree to which the teaching task includes reading texts that use and develop 
academic understanding and vocabulary, and offer opportunities for multiple interpretations and 


higher-order thinking. 


The remaining four dimensions of the revised assignment measure address the question, “what 
makes a great module?” The dimensions capture the extent to which teachers engage in high quality 
instructional strategies, and address the four stages of instructional practice delineated in the LDC 
Instructional Ladder. These dimensions also provide an opportunity to assess the coherence and clarity of 


the module as a whole: 


6) Fidelity to LDC Module Instruction is the degree to which module instruction, activities, and 
the teaching task address each of the four stages of instructional practice (preparation for the task, 
reading process, transition to writing, writing process). The main sources of information include 
the instruction section of the Module Creator handout and any information provided about the 


distribution of activities and time spent on each of the four stages. 


7) Quality Instructional Strategies considers the degree to which the module provides clear 
instructional strategies aimed at helping students develop literacy skills and successfully complete 
the teaching task, and the degree to which module instruction and activities scaffold critical 
thinking and performance in a way that is meaningful within the context of a given field or 
subject-matter. In addition to the instruction section of Module Creator and completed teacher 
logs, evidence for this dimension can be found in classroom handouts and examples of student 


work. 


8) Coherence and Clarity of Module is the degree to which there is logical alignment between 


the teaching task and other module goals with readings, mini-tasks, and instructional strategies. 
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All classroom artifacts should be used to assess this and the final dimension, including materials 
and information listed or uploaded into Module Creator, classroom handouts, and examples of 


student work. 


9) Overall Impression is a holistic assessment of LDC Module. This dimension gives raters the 
opportunity to make an overall assessment of LDC implementation, and determine the extent to 
which a module contributes to student college readiness and development of advanced literacy 


skills. 


Methods 

LDC Modules were collected from 8" grade social studies and science teachers who were part of 
a larger quasi-experimental study of LDC implementation. We asked that teachers submit data on two 
modules if possible, preferably in the fall and spring semesters. Most teachers taught at least two 
modules, but some taught only one. Although timing of module submission varied, in all cases Module 2 
reflected instruction that occurred later in the academic year. Each module captured teacher LDC 
instruction over the course of 2-4 weeks. As part of the larger study, teachers were also asked to 
complete an online log twice per week during each week of LDC instruction. These online logs were 
designed on the online survey engine Qualtrics and we requested that teachers submit modules and 
classroom artifacts online when they completed logging on a given module. In addition to the specific 
instructional materials listed above, teachers had an opportunity to submit any additional materials that 
they believed would help us understand their module instructional practice. We followed up with teachers 
who had completed logging but did not submit their materials. The final sample included 21 social 
teachers, who submitted 40 social studies modules, and 17 science teachers who submitted 29 science 
modules. 

We used a within-subject design to explore teacher variation in LDC implementation. The design 
allowed us to investigate variation in LDC implementation over time, as well as between subject areas 


(science and social studies) and between states (Pennsylvania and Kentucky). 


Pilot Study 

We piloted the assignment measure and rater training in April 2013 with two expert teachers, 
including one science and one social studies teacher. The four science and social studies modules used 
for the pilot session were selected from a pool of completed modules that were submitted to our research 
team from study participants in the early spring. These modules included all of the classroom materials 


that teachers were asked to submit, and represented the types of teaching task topics similar to what raters 
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encountered in the official rating session, including those on climate change, transportation of hazardous 
materials, the Electoral College, and the American Revolution. Both raters found the dimensions logical 
and intuitive, and the training in LDC, Module Creator, and the CCSS thorough and useful for their own 
practice. Their suggestions for how we might clarify the rubric and rater training protocol were minor 
and almost all were incorporated. We determined that it took approximately 45 — 60 minutes for a teacher 


to rate a module. 


Recruitment and Training 

We recruited raters for a week-long scoring session held on June 17-21, 2013. We generated a 
list of eligible and potentially interested teachers from local district leaders and experts in social studies 
and science education and we ultimately invited over 90 teachers to apply. Five social studies and four 
science teachers were selected from a total pool of over 70 applicants. Raters were offered $200 for each 


day of participation, in addition to breakfast, lunch and parking. The recruitment letter explained: 


UCLA's Center for Research on Evaluation, Standards, & Student Testing (CRESST) and the 
Gates Foundation are investigating an instructional intervention aligned to the Common Core 
State Standards (CCSS) in 8th grade science and social studies classrooms. We are looking for 
experienced teachers to review and score instructional materials collected in middle school 
science and social studies classrooms during the 2012-2013 school year. Rating sessions will 
take place from Monday, June 17, 2013 - Friday, June 21, 2013. 


Job Details: 

- Raters will received a daily stipend of $200 ($25/hour) 

- Breakfast, lunch, and parking will be provided 

- Sessions will take place at the UCLA campus from 9:00AM-5:00PM 


Job Qualifications: 
- Experienced teachers who currently or recently taught 8th grade science and social studies 
- May not have been a UC Employee within the last two years 


If you are interested in rating instructional materials or would like further information about our 
study, please fill out and return the attached form April 30, 2013. 


Rater training and calibration occupied the first day and a half of the weeklong session. The first 
morning was devoted to introducing raters to LDC and Module Creator, as well as to familiarizing them 
with the dimensions of the rubric. During the afternoon of the first day and the morning of the next, 
social studies and science teachers independently scored two anchor modules in their subject areas. Each 


group of teachers met with a subject area expert who had also scored the modules to discuss and calibrate 


their ratings. The remainder of the week was devoted to rating modules. Each teacher rated 


approximately 24 modules. 


Generalizability Study Design 

Generalizability theory, or G theory, is a statistical framework for determining the 
reliability of measurements under specific conditions. The theory asserts that there are multiple 
sources of error, rather than a single error term as in classical reliability theory. Each source of 
potential error is considered a facet, and the goal of a G-study is to determine the amount of error 
caused by each facet and the interaction of facets. We conducted Generalizability studies to 
investigate the reliability of module ratings, with the goal of separating true teacher variation 
from other sources of measurement error, for example rater variation or variation in LDC 
implementation over time (Shavelson & Webb, 1991). We also conducted a decision study, or D 
study, to estimate how generalizability coefficients would change if different aspects of the study 
(e.g., number of raters, number of modules) were altered. Therefore, for each set of modules, we 
estimated two kinds of reliability coefficients: a generalizability coefficient (p) reflecting 
consistency in relative score interpretations (i.e., rank ordering) and a dependability coefficient 
(>) for absolute interpretations (i.e., judging performance against set criteria or standards). In 
both cases, we estimated hypothetical scenarios that vary the number of modules collected per 
teacher and the number of raters. Finally, we conducted exploratory factor analyses to investigate 
the extent to which one or more dominant factors or traits underlie the correlation patterns 
observed among ratings on the nine rubric dimensions. 

Social studies and science modules were analyzed separately because raters exclusively scored 
modules in their subject area. A fully crossed design with all raters scoring all modules in a given subject 
area was not feasible due to time constraints, but most raters scores majority of the teacher artifacts. We 
analyzed all valid data by using the missing data option in SPSS, which enabled us to omit an observation 


when there were missing values in the independent effects. 


Qualitative Data 
In addition to the analyses described above, we conducted 20-30 minute interviews with 


individual raters during the final two days of week to inquire about their experience with the assignment 
measure and their initial reactions to teacher implementation of LDC. The nine raters also completed a 


short survey on Qualtrics that asked them to review their experience over the course of the week. The 


survey was comprised of three sections. The first asked raters to “indicate how strongly you agree or 
disagree with the following statement as it applies to each rubric dimension:” 
1) Iclearly understood this dimension and the aspect of LDC module instruction that it was intended 
to capture. 
2) The LDC module notebook provides sufficient evidence to judge this aspect of LDC module 
instruction. 
3) Iam confident about the ratings I assigned in this dimension. 
In the second section, the raters were asked to indicate how useful each of the following components were 
“as a source of information for judging each dimension:” one-page information form; module creator 
handout; other planning materials; reading/texts; reading supports; writing supports; samples of student 
work; other instructional materials. Finally, raters were asked to comment on what knowledge, 


experiences, or personality characteristics might make someone a good rater of LDC modules. 


Results 
Table 1 presents mean scores for all social studies and science modules across all dimensions. 


Several cautions must be considered in reviewing these findings. First, given the limited sample size, we 
must be careful of making generalizations about social studies or science LDC implementation. It is 
important to note that different groups of raters scored each set of modules, and thus the scores are 
incomparable. Moreover, it is conceivable that science raters may have been more lenient in their scoring 
for any number of reasons, including their lack of experience integrating literacy and content. 

However, we see that, in general, science modules received higher ratings across dimensions, 
with the exception of two dimensions. This may appear counter-intuitive: one would think that social 
studies teachers would have an easier time integrating writing instruction, given the subject matter. The 
mean scores may suggest science teachers were better able to integrate subject matter content and literacy 
within the framework of LDC instruction. Science teachers were most successful at following and 
elaborating upon the structure of the LDC module. Their modules not only contained the four “skill 
clusters” —Preparing for the Task, Reading Process, Transition to Writing, and Writing Process—but 
these skills clusters included mini-tasks that went beyond the default options provided on Module Creator. 
Science teachers were also able to find texts that aligned with the writing task, but these were not 
particularly rigorous, in that they did not necessarily promote multiple interpretations or higher-order 
thinking. 

Social studies modules scored higher than science modules on the Effective Writing Task (the 
linchpin of the module) and Text Rigor (the academic substance of the texts) suggesting that social 


studies teachers struggled less with integrating writing tasks into their content instruction, and more with 
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designing instruction and supporting student execution of the task. Of all the dimensions, social studies 
teachers were most successful at finding texts to align with the writing task. Again, it is possible that 
social studies raters were more exacting in their ratings, given their experience integrating literacy and 
content instruction. For the remainder of the report, we present results for social studies and science 


modules separately. 


Table 1: Mean Scores for Social Studies and Science Modules Across Dimensions (Scale 1-5) 


Dimensions Social Studies Science 
1. Effective Writing Task 3.40 3.05 
2. Alignment to Literacy and Content Standards 2.42 3.25 
3. Text Alignment 3.44 3.52 
4. Text Appropriateness 3.24 3.41 
5. Text Rigor 3.35 3.05 
6. Fidelity to LDC Module Instruction 3.04 3.85 
7. Quality Instructional Strategies 2.85 3.09 
8. Coherence and Clarity of Module 2.82 3.23 
8. Overall Impression 2.73 3.04 
Social Studies Modules 


Descriptive Statistics by Module and State: Social Studies 

Table 2 presents descriptive statistics for social studies by dimension separated by module. A 
few comparative observations can be made. First, surprisingly, the average rating for the second social 
studies module was lower than the first in almost all dimensions except Overall Impression (D9) and 
Effective Writing Task (D1), where the improvement from the first to the second module was slight. It is 
unclear why social studies modules would have, on average, declined over the course of the year, and it is 
important to consider that ratings could reflect teacher fatigue (and a lack of willingness to upload 
relevant, supplementary instructional materials). The case for teacher fatigue may be further supported by 
the consistent means for Overall Impression and Effective Writing Task between Modules 1 and 2; these 
two dimensions rely less on the presence of extensive artifacts and more on raters’ gut sense of the 
intellectual value and rigor of the instruction. That the means on D1 and D9 remained consistent suggests 


that module quality may not have varied as much as may appear at first glance. 


Table 2: Descriptive Statistics by Dimension and Module for Social Studies Teachers 
(N=40) 


Module 1 Module 2 
(N=19) (N=21) 
Dimensions M SD M SD 
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Effective Writing Task 3.37 1.08 3.39 1.18 


Alignment to Literacy and Content Standards 2.54 1.25 2.22 1.02 
Text Alignment 3.49 1.07 3.37 1.31 
Text Appropriateness 3.32 0.78 3.15 1.11 
Text Rigor 3.51 1.09 3.17 1.26 
Fidelity to LDC Module Instruction 3.19 1.04 2.80 1.18 
Quality Instructional Strategies 2.88 1.05 2.79 0.99 
Coherence and Clarity of Module 2.83 1.15 ZA 1.11 
Overall Impression 2.70 0.93 2.74 1.02 


Tables 3 present descriptive statistics for social studies modules by state.' Again, we must be 
cautious about drawing conclusions from these results, as differences may be a sign of pre-existing 
differences between teachers. Nonetheless, we see here that scores on social studies modules did not vary 
tremendously by state. Perhaps the biggest distinction is that teachers in IU13 scored considerably higher 
on Fidelity to LDC Module Instruction (D6). The higher score on this dimension likely reflects the fact 
that all teachers from I[U13 submitted modules designed on the online LDC platform, Module Creator. 
Approximately half of Kentucky social studies modules were submitted using an older paper template for 
LDC or without any template. If it was difficult for raters to identify the four skill clusters —Preparing for 
the Task, Reading Process, Transition to Writing, and Writing Process—the module could not receive a 
score of 3 in D6. However it is important to note that a module could earn a 3 on D6 by simply 
defaulting to mini-task options automatically provided in Module Creator for each skill-cluster. In other 
words, the average score of 3.35 in D6 for IU13 social studies modules does not necessarily suggest that 
those teachers elaborated or expanded on the default options provided for instruction. The overall mean 
across dimensions for IU13 social studies modules (M=3.03) is only marginally higher than that for 


Kentucky (M=2.99). 


Table 3: Descriptive Statistics for Social Studies Modules by Dimension and State 


(N=40) 
Kentucky 
IU13 (N=18) (N=22) 
Dimensions M SD M SD 
Effective Writing Task 3.38 0.95 3.38 1.26 
Alignment to Literacy and Content Standards 2.71 0.99 2.10 1.19 
Text Alignment 3.36 =1.09 3.48 1.28 
Text Appropriateness 3.07 0.86 3.36 1.03 
Text Rigor 3.02 1.03 3.58 1.25 
Fidelity to LDC Module Instruction 3.35 0.64 2.70 1.33 
Quality Instructional Strategies 2.80 0.87 2.86 1.13 


1 See Appendix B for comparisons with scores on five LDC modules designed at the district level in 
Hillsborough, FL. 
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Coherence and Clarity of Module 2.80 0.85 2.80 1.31 
Overall Impression 2.76 0.90 2.68 1.04 


Generalizability Study: Social Studies 

Table 4 presents the estimated variance components for a teacher by rater by module (t*r*m) 
generalizability analysis for social studies modules. Again, the goal here is to separate true teacher 
variation from other sources of measurement error, for example rater variation or variation in LDC 
implementation over time. This model partitions variance into seven components; each column represents 
a source of variation in module ratings on each of the rubric dimensions. In this particular G study, if we 
found high systematic variation between raters that might suggest that the ratings were not reliable. On 
the other hand, we might expect to find high variation across modules, because teachers may improve at 
LDC implementation over time. The main effects reflect true variance across teachers (o’t) and error 
variance across raters and modules (o’r, o’m); a residual term (o’trm,e) combines the t*r*m interaction 
and residual error unexplained in the model. Across the dimensions, the results are quite promising: we 
see virtually no variation across raters, and high variation between teachers. In particular, it is important 
to note the high teacher variance captured in D9: Overall Impression, a dimension that asked raters to 
assess the degree to which the module contributed to student college readiness and development of 
advanced literacy skills. This suggests that raters were able to assess the overall instructional potential of 
the modules, separate from the particularities of LDC implementation. 

For the three dimensions where we see lower variation between teachers (D1: Effective Writing 
Task; D3: Text Alignment; and D4: Text Appropriateness) we nonetheless see that a large portion of the 
variance for those dimensions is captured in the teacher by module interaction effect. The o’tm interaction 
suggests that certain teachers’ scores on these dimensions varied between their first and second module. 
In other words, although we do not see high variation overall between first and second modules (om), it 
appears that differences between modules were tied to particular teachers. As discussed earlier, this 
variation may reflect inconsistencies in how teachers assembled and uploaded module materials, or it may 
reflect true variation in teacher implementation of LDC. In either case, this variation suggests that any 
single module may not accurately represent teacher LDC implementation. Finally, the residual error term 
(o’trm,e) may also reflect systematic rater inconsistency and other sources of error not captured in the 
design. 


Table 4: Generalizability Studies of Social Studies Module Ratings (t*r*m*tr*tm*rm)" 


Percentage of Total Variance (%) 
Dimension o2t o2r o2m o2tr o2tm o2rm o2trm,e 
Effective Writing Task 5.6 0.0 0.0 11.3 22.2, 6.0 55.0. 
Alignment to Standards 42.2 3.1 1.8 0.0 16.9 2.5 33.5 
Text Alignment 15.1 1.8 0.0 2.3 ee 0.0 45.5" 
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Text Appropriateness 14.6 2.1 0.0 3.2 23.0. 1.3 55.9. 
Text Rigor 33.1 | oo | 23 0.0 | 24.5 0.0 | 40.1. 
Fidelity to LDC Module Instruction 49.4, 0.0 3.7 0.0 25. 0.0 24.2 
Quality Instructional Strategies 32.7. 0.0 0.0 12.6 21.8. 0.1 32.9. 
Coherence and Clarity of Module 40.2. 0.0 0.0 15.5 12.0 4.5 27.8. 
Overall Impression 34.8. 2.6 0.0 7.2 17.1 1.7 36.5. 


“Indicates large proportion of variance captured by this facet. 


Decision Study: Social Studies 

We also conducted decision studies to determine dependability estimates under hypothetical 
scenarios that varied the number of modules and raters. Dependability estimates provide information 
about the consistency of absolute performance (in this case, on a given dimension) independent of others’ 
performance, rather than consistency of relative standing. These findings are presented in Table 5. In 
social studies, with two modules, estimated dependability with 3 raters exceeds .60 for all dimensions 
except Effective Writing Task, Text Alignment, and Text Appropriateness. The estimates for all three 
dimensions are slightly improved with 3 modules, but still below .5. These are the same three dimensions 
for which we saw low teacher variance, which may suggest that the dimensions are simply not effective in 
discriminating among teachers. On the other hand, considering that the teacher by module variance is 
quite large for all three of these dimensions, it is possible that the low dependability estimates for these 


dimensions raises questions about how many modules might be necessary to capture true teacher variation. 


Table 5: Social Studies: Decision Studies of Module Ratings by Dimension (t*r*m Design) 
Dependability Coefficients 


2 Modules 3 Modules 
Raters (Crossed) _ Raters (Crossed) 
Dimension 2 3 2 3 
Effective Writing Task 0.149 0.182 0.194 0.237 
Alignment to Literacy and Content Standards 0.680 0.721 ~—-0.754—-0.790" 
Text Alignment 0.327 0.362 0.413 0.453 
Text Appropriateness 0.339 0.390 0.423 0.480 
Text Rigor 0.585 0.622 0.679 0.712" 
Fidelity to LDC Module Instruction 0.719 0.741 0.794 0.811 
Quality Instructional Strategies 0.562 0.614 0.632 0.684" 
Coherence and Clarity of Module 0.648 0.709 -0.701.——«0.759" 
Overall Impression 0.601. 0.656 0.672.——:0.724" 


“Indicates dependability estimates greater than .60. 


Factor Analysis: Social Studies 
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Table 6 presents the principal component solutions extracted from the average teacher scores for 
each dimension over raters and modules. Overall Impression (D9) was initially excluded to avoid 
artificial unidimensionality in the data, but later included when we determined that it did not skew the 
loadings. The result of the factor analysis for social studies module ratings is encouraging: all 9 
dimensions load heavily on one factor, and account for 63% of the variance in social studies ratings. This 
suggests a dominant factor or trait underlying the nine dimensions of LDC implementation measured in 
the Assignment Measure, and makes the case for the coherence and conceptual validity of the tool. It is 
interesting to note that two of the dimensions (Alignment to Literacy and Content Standards: D2 and 
Fidelity to LDC Implementation: D6) load equally well on a second factor. Because both of these 
dimensions require the rater to evaluate the module according to outside criteria (e.g., standards, LDC 
framework), it is possible that they point to a trait that is distinct from how the rater might go about 


evaluating classroom instruction. 


Table 6: Principal Component Analysis of Social Studies Modules (N=40) 


Component 
1 2 
D1_mean .736 -.191 
D2_mean .604 .656 
D3_mean .803 -.415 
D4 mean 827 -.351 
D5_mean .768 -.389 
D6_mean .664 599 
D7_mean .906 119 
D8 mean 948 142 
D9_mean 927 029 
Extraction Method: Principal Component 


Analysis. 


Science Modules 
Descriptive Statistics by Module and State: Science 
Table 7 presents descriptive statistics for science by dimension separated by module. We found 
that with the exception of DS: Text Rigor, average ratings improved across all dimensions between 
Modules 1 and 2. This trend reflects what we would expect as teachers become more experienced in 


implementing LDC instruction. 


Table 7: Descriptive Statistics by Dimension and Module for Science Teachers (V=29) 
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Module 1 Module 2 


(N=15) (N=14) 

Dimensions M SD M SD 

Effective Writing Task 3.00 1.20 3.09 1.14 
Alignment to Literacy and Content Standards 2.98 1.24 3.45 1.06 
Text Alignment 3.46 1.24 3.55 1.14 
Text Appropriateness 3.30 1.18 3.53 1.06 
Text Rigor 3.06 1.29 3.00 1.20 
Fidelity to LDC Module Instruction 3.59 0.94 4.04 0.88 
Quality Instructional Strategies 3.04 1.18 3.09 1.08 
Coherence and Clarity of Module 3.17 1.22 3.23 1.16 
Overall Impression 2.85 1.14 3.19 1.08 


Tables 8 present descriptive statistics for science modules by state. Science modules in Kentucky 
(M=3.44) were consistently higher across dimensions than those from IU13 (M@=3.05). It is important 
that we refrain from overstating these mean differences — all fall well within the standard deviations. 
Furthermore, observed differences could be the result of pre-existing differences between teachers. 
Nonetheless, given the consistency of these differences, it is worth exploring and comparing the 
professional development offered to [U13 and Kentucky science teachers, to identify whether and/or how 


Kentucky teachers were able to develop stronger modules. 


Table 8: Descriptive Statistics for Science Modules by Dimension and State (V=29) 


Kentucky 
IU13 (N=14) (N=15) 

Dimensions M SD M SD 

Effective Writing Task 2.80 1.15 3.27 1.14 
Alignment to Literacy and Content Standards 3.18 1.11 3.21 1.24 
Text Alignment 3.39 1.10 3.62 1.27 
Text Appropriateness 3.10 1.14 3.69 1.04 
Text Rigor 2.80 1.08 3.25 1.36 
Fidelity to LDC Module Instruction 3.59 0.79 4.00 1.03 
Quality Instructional Strategies 2.92 1.00 3.19 1.24 
Coherence and Clarity of Module 2.92 1.10 3.46 1.23 
Overall Impression 2.76 0.99 3.25 1.19 


Generalizability Study: Science 

Table 9 presents the estimated variance components for a teacher by rater by module (t*r*m) 
generalizability analysis for science modules. This model partitions variance into seven components. The 
main effects reflect true variance across teachers (ot) and error variance across raters and modules (0°r, 
om); a residual term (o°trm,e) combines the t*r*m interaction and residual error unexplained in the 


model. The results here resemble those found for social studies module ratings, with a few differences. 
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Although we again see virtually no variance across raters, we do see high variance in the teacher by rater 
interaction for two of the dimensions: Text Alignment (D3) and Text Rigor (D5). And although we see 
high variation between teachers for four of the dimensions (D1, D6, D7, D9), the variance components for 
the remaining dimensions are fairly low. 

How do we interpret these results? First, it is important to note once again the high teacher 
variance captured in D9: Overall Impression, a dimension that asked raters to assess the degree to which 
the module contributed to student college readiness and development of advanced literacy skills. This 
suggests that raters were able to assess the overall instructional potential of the modules, separate from 
the particularities of LDC implementation. On the other hand, the high variance in the rater by teacher 
interaction (o°tr) signals inconsistencies in rater understanding or use of the scoring rubrics with different 
teachers. That we see high variation for this interaction in two of the dimensions dealing with text 
selection (D3 and D5), suggests that science raters were less clear—or possibly systematically 
disagreed—about what constituted a relevant and academically challenging text in science. 

As for the five dimensions for which we see low variation between teachers (D2: Alignment to 
Content and Literacy Standards; D3: Text Alignment; D4: Text Appropriateness; D5: Text Rigor; and 
D8: Coherence and Clarity), we nonetheless see that a large portion of the variance for those dimensions 
is captured by the teacher by module interaction effect. The o’tm interaction suggests that differences 
between modules were tied to particular teachers. As discussed earlier, this variation may reflect 
inconsistencies in how teachers assembled and uploaded module materials, or it may reflect true variation 
in teacher implementation of LDC. In either case, this variation suggests that any single module may not 
accurately represent teacher LDC implementation. Again, the variance captured by residual error term 
(o’trm,e) remains high for all but one dimension, likely reflecting systematic rater inconsistency and other 


sources of error not captured in the design. 


Table 9: Generalizability Studies of Science Module Ratings (t*r*m*tr*tm*1rm) 


Percentage of Total Variance (%) 

Dimension o2t o2r o2m o2tr o2tm o2rm o2trm,e 
Effective Writing Task 24.8 0.0 0.0 3.7 6.8 11.3 53.4, 
Alignment to Standards 17.6 0.0 3.1 17.6 35.2. 4.0 22.5" 
Text Alignment 71] 02] oo | 254 | 412° 22 | 23.9" 
Text Appropriateness 9.1 8.0 0.0 18.7 25.8. 0.4 38.1. 
Text Rigor 12.6 0.0 0.0 28.7, 40.0. His 11.5 
Fidelity to LDC Module Instruction 33.8. 14.0 8.7 2.3 0.0 0.6 40.6 
Quality Instructional Strategies 34.9" 0.0 0.0 15.3 0.0 6.8 43.0. 
Coherence and Clarity of Module 15.0 0.0 0.0 6.4 21.0. 7.2 50.4 
Overall Impression 27.9 0.0 0.0 17.1 4.8 14.2 36.0 


“Indicates large proportion of variance captured by this facet. 
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Decision Study: Science 

We also conducted decision studies to determine dependability estimates under hypothetical 
scenarios that varied the number of modules and raters. These findings are presented in Table 10. In 
science, estimated dependability with 2 modules and 3 raters is quite low for five of the nine dimensions, 
but exceeds .60 in Effective Writing Task, Fidelity to LDC Instruction, Quality Instructional Practices, 
and Overall Impression. That these four dimensions all deal with the general sense of the module, rather 
than the particularities of text selection, text preparation, and standards alignment, might suggest that 
science raters could reliably discern teacher overall fidelity of implementation, but were less confident 
about their interpretation of the specific components of module design. None of the dimensions are 
substantially improved by adding another module to the model. These findings of the decision studies are 
considerably less robust than what we saw with social studies, which might be a consequence of the 
smaller sample size. However, it also may suggest that raters of science modules may need additional 
training in LDC, and perhaps a better understanding of what the effective integration of literacy 


instruction and science content looks like. 


Table 10: Science: Decision Studies of Module Ratings by Dimension (t*r*m Design) 


Dependability Coefficients 


2 Modules 3 Modules 
Raters Raters (Crossed) 

Dimension 2 3 2 3 
Effective Writing Task 0.536 0.616 0.624 0.698 
Alignment to Literacy and Content Standards 0.338 0.375 0.405 0.450 
Text Alignment 0.150 0.174 0.186 0.219 
Text Appropriateness 0.202 0.244 0.243 0.294 
Text Rigor 0.244 0.278 0.290 0.335 
Fidelity to LDC Module Instruction 0.597 0.670" 0.654 0.724" 
Quality Instructional Strategies 0.653 0.723 0.687 0.767" 
Coherence and Clarity of Module 0.347 0402 0.430 0.491 
Overall Impression 0.543 0.629. 0.601 0.684" 


“Indicates dependability estimates greater than .60. 


Factor Analysis: Science 

Finally, Table 11 presents the principal component solutions extracted from the average scores 
for each dimension over raters and modules. Overall Impression (D9) was initially excluded to avoid 
artificial unidimensionality in the data, but later included when we determined that it did not skew the 
loadings. The result of the factor analysis for science module ratings is encouraging: all 9 dimensions 


load heavily on one factor, and account for 67% of the variance in social studies ratings. This suggests a 
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dominant factor or trait underlying the nine dimensions of LDC implementation measured in the 


Assignment Measure, and makes the case for the coherence and conceptual validity of the tool. 


Table 11: Principal Component Analysis of Science Modules (N=29) 


Component 

1 2 
D1_mean .706 -.537 
D2_mean 658 -.142 
D3_mean .896 -.194 
D4 mean .833 -.329 
D5_mean 919 -.241 
D6_mean 551 759 
D7_mean .884 355 
D8 mean 877 335 
D9_mean 955 121 
Extraction Method: Principal Component 
Analysis. 


Qualitative Results 

The qualitative data collected from the rater interviews and the rater survey sheds light on how 
raters experienced the scoring session and suggests ways to potentially improve rater reliability in the 
future. One key finding that emerges from analysis of qualitative data is the sense that the artifacts 
collected and assembled in each module notebook were not necessarily sufficient to assess or make 
inferences about LDC instruction. Both science and social studies teachers found this to be especially 
true for dimensions 6, 7, and 8 (Fidelity to LDC Implementation, Quality Instructional Strategies, and 
Coherence and Clarity). When asked to assess the relative strength of each source of information, both 
social studies and science teachers found the actual Module Creator print-out not useful or only somewhat 
useful for rating most of the dimensions, with the exception of Fidelity to Implementation (D6) and 
Alignment to Standards (D2), two dimensions that required teachers to consult specific sections of the 
print-out. But many teachers did not find it particularly useful in rating the text-related dimensions, 
Instructional Quality, Coherence and Clarity, and the Overall Impression. 

By contrast, both groups of raters found the actual readings and texts extremely useful in 
assessing most of the dimensions of the rubric, especially, not surprisingly, the text-related dimensions. 
Social studies raters also found the texts useful for rating Quality Instructional Practices, Coherence and 
Clarity, and Overall Impression. Interestingly, science teachers reported that the texts were not useful in 


making such inferences. Likewise, whereas social studies raters found Reading Supports extremely 
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useful in rating almost all dimensions, science raters mostly found them useful in rating D6, D7, D8, and 
D9, but Not Useful for rating D1-D5. At the risk of making too much of this finding, it is possible that 
science raters were less comfortable drawing inferences about literacy instruction from the assembled 
artifacts than social studies raters. This finding is supported by our impression from the interviews that 
science teachers had less experience teaching literacy than social studies teachers. 

Raters suggested certain sources of information that would have made scoring easier. First, one 
rater suggested that in addition to final student work, teachers submit evidence of student progress in the 
form of revised drafts. Several raters wished they had had more evidence of teacher implementation — 
lesson plans, scaffolds, and actual readings, for those modules where teachers just listed readings on 
Module Creator. Raters also wanted much more information about school context, student demographics, 
and most importantly, the extent of professional development and support that teachers received in 
implementing LDC. Given the extent to which LDC departs from business-as-usual in content classrooms, 
raters felt that knowing the extent and quality of teacher training in the approach would have informed 
their evaluation. 

We also asked raters to give their impressions of LDC as an intervention geared to helping 
teachers integrate literacy and content instruction. The majority of raters indicated support and 
enthusiasm for the intervention, but highlighted (as mentioned above) what they saw as a tremendous 
need for professional development and coaching. One rater warned that LDC should not be seen as a 
“magic bullet.” Several raters questioned whether the stronger modules reflected the strength of LDC as 
an intervention or the instructional skills that the teacher brought to the model. Raters emphasized that 
the strong modules basically reflected good instruction, and one rater thought that in some modules “LDC 
gets in the way.” Science raters, in particular, highlighted that science teachers will need much more help 
in teaching reading and writing, beyond the LDC template. At the same time, science raters were quite 
enthusiastic about the actual topics that they saw in the modules, and several stated that the modules gave 
them ideas for their own practice. Social studies teachers were less enthusiastic about the actual modules 
they rated. Several commented on the lack of differentiation and the lack of instructional support that 
pushed students to consider multiple perspectives and read text critically. Others were frustrated that the 
rubric did not include a dimension to rate the quality of the actual content delivered in the module. For 
example, they found instances where student work that was marked ‘high’ by the teacher included glaring 


historical inaccuracies. 


Discussion 
Overall, we are buoyed by the findings discussed above and believe the CRESST Assignment 


Measure to be a promising assessment tool for gauging teacher success in integrating literacy and content 
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instruction. In both the social studies and science analyses, we found low rater variance and high teacher 
(or teacher by module) variation. These findings indicate that raters generally found the dimensions 
intuitive and aligned with the available sources of information. Moreover, the factor analyses indicate 
that all dimensions load on a single factor, making the case that the CRESST Assignment Measure 
effectively measures a coherent trait that might be understood to be LDC implementation, or perhaps 
more generally, instructional quality in the integration of literacy and content. These findings are 
especially promising given our limited data set and the myriad logistical challenges of artifact collection. 

At the same time, we can identity certain questions that are worthy of further investigation and 
consideration. First, how much instructional material is required for raters to make informed, reasonable 
inferences about LDC implementation? Raters would ideally have additional artifacts on which to base 
their judgments, however requiring teachers to submit additional artifacts raises logistical considerations 
in terms of efficiency and cost-effectiveness. Nonetheless, if the tool is to be used in future evaluations of 
LDC instruction, it is worth exploring whether additional samples of classroom instruction (e.g., 
classroom handouts, actual lesson plans, and samples of all student work on mini-tasks) would increase 
rater reliability. Second, in both the social studies and science analyses, we saw considerable teacher by 
module variation. Clearly, a single module is not sufficient to achieve a high dependability estimate; 
evidence from the decision study suggest that 3 modules may be sufficient in social studies, but perhaps 
not in science. With teachers struggling to understand how to implement LDC, perhaps it is not 
surprising that quality varies considerably across modules. If that’s the case, it may be unreasonable to 
expect that we can get a reliable teacher-level score with a small sample size of modules. This, too, needs 
to be explored if the CRESST Assignment Measure is to be used in further evaluation of LDC instruction. 

Third, we found that the results for the science module ratings to be slightly less robust than those 
for social studies. It is unclear whether this is a result of smaller sample size, or whether science raters, in 
general, were less experienced and less familiar with instruction that integrates literacy and content. We 
suspect the latter based on our qualitative data and if so, there are implications not only for rater training, 
but also for how to support science teachers implementing LDC. It is particularly telling that the less 
robust findings in the science module analyses tended to cluster around the text-related dimensions (D3, 
D4, D5). These findings might suggest that science teachers (and raters) need additional support in 
identifying and evaluating texts that can be used in LDC implementation. 

Lastly, there are some indications in both the quantitative and qualitative data that the text-related 
dimensions might not be sufficiently distinct, or sufficiently discriminating. In future iterations it is worth 
exploring whether they might be collapsed. We only caution that this not be done prematurely. The three 
dimensions are designed to capture distinct aspects of module design —the selection of texts that align 


with the content demands of the template task (D3); teacher attention to grade appropriate reading levels 


PAL 


(D4); and teacher attention to the disciplinary or academic rigor of the reading (D5). Admittedly, at first 
glance these distinctions may appear subtle. However, we believe that they capture some of the nuance 
involved in developing effective and teachable modules of LDC instruction. 

How one chooses to revise the CRESST Assignment Measure no doubt depends on how it will be 
used. We see potential for the tool both in future evaluations of the program, as well as in professional 
development. If used for professional development purposes, we believe that more nuanced dimensions 
(e.g., the text dimensions described above) can be useful in helping teachers hone the skills necessary in 
designing instruction around texts. We might even suggest adding additional dimensions that assess 
reading instruction separately from writing instruction. Clearly, such hair-splitting might not be 
necessary in a large-scale evaluation. In both cases, however, we see great potential for the tool in 


helping content teachers design quality instruction that integrates Common Core State Standards. 


Ze 
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Appendix A 


Each dimension is scored on a 5-point scale ranging from “Fully Present or Realized” to “Not Present or 
Realized.” 


Fully Sufficiently Moderately Barely Not 
Present or Present or Present or Present or Present or 
Realized Realized Realized Realized Realized 
5 4 3 2 1 


IMPORTANT: Descriptions are provided for three anchor points in the scale: 5 (Fully Present or Realized), 
3 (Moderately Present or Realized), and 1 (Not Present or Realized). Use the intermediate points in the 
scale (4 and 2) to rate assessment practice that lies between 5 and 3 and 3 and 1. 


Dimension 1: Effective Writing Task 

Dimension 2: Alignment to the CCSS and Local and State Literacy and Content Standards 
Dimension 3: Text Alignment 

Dimension 4: Text Appropriateness 

Dimension 5: Text Rigor 

Dimension 6: Fidelity to LDC Module Instruction 

Dimension 7: Quality Instructional Strategies 

Dimension 8: Coherence and Clarity of Module 

Dimension 9: Overall Impression 


Dimension 1: Effective Writing Task 


Definition: Degree to which teaching task makes effective use of the template task’s writing mode (i.e., 
argumentation or explanation); requires sustained writing and effective use of ideas and evidence to substantiate 
claims; and is feasible for most students to complete (i.e., appropriate for the grade-level and subject matter). 


Main Sources of Information: 

Module Creator Handout (Task) 

- Read and evaluate the teaching task, student background/prior knowledge, and summary information. 
- Evaluate the difficulty or ease students may encounter trying to answer the question. 

- Compare module teaching task to teaching task template options. 


5. Fully Realized The teaching task and performance expectations for the module are 
explicit and clear, require students to engage in higher-order thinking 
and writing, and are appropriate for the grade-level and subject 


matter. 

4. 

3. Moderately Present or Realized Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher-order 
thinking and writing and/or are not appropriate for the grade-level 
and subject matter. 

2. 

1. Not Present or Realized Minimal evidence of an effort to identify explicit and clear teaching 


task and performance expectations that provide opportunities for 
critical thinkéhg and are appropriate for the grade-level and subject 
matter. 


Dimension 2: Alignment to the CCSS and Local and State Literacy and Content Standards 


Definition: Extent to which module addresses content essential to the discipline, as well as reading comprehension 
and writing standards informed by local and state standards. 


Main Sources of Information: 

Module Creator Handout (Task) 

- Read and evaluate the standards included in the module. 

- Module should include ELA as well as subject matter CCSS/state standards. 

- Compare and contrast the standards the module includes with those that could have been included. 

- Particular attention to content standards (CCSS History/Social Studies, Science, and Technical Subjects); State 
Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 


5. Fully Realized Module specifically addresses content essential to CCSS and local or 
state standards in science or social studies, as well as reading 
comprehension and writing. All standards are well aligned to the topic 
and teaching task. 


4. 

3. Moderately Present or Realized Module broadly addresses content essential to CCSS and local or state 
standards in science or social studies and reading comprehension and 
writing. Standards are sufficiently aligned to the topic and teaching 
task. 

2. 

1. Not Present or Realized Minimal evidence that module addresses content essential to the 


discipline and literacy standards. Standards are poorly aligned to the 
topic and teaching task. 


Dimension 3: Text Alignment 


Definition: Degree to which assigned texts address teaching task content. 


Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 
- Read and evaluate texts (hard copies or online). 
Student Work 

- References in student work. 


5. Fully Realized Assigned readings address the disciplinary content in science or social 
studies and give students the opportunity to gather information needed 
to complete the task. Readings are well aligned to the topic and 
teaching task, and provide students with well-balanced perspectives. 


4. 

3. Moderately Present or Realized Assigned readings mostly address the disciplinary content in science or 
social studies and give students some opportunities to gather 
information needed to complete the task. Readings are sufficiently 
aligned to the topic and teaching task, and provide students with 
moderately balanced perspectives. 

2. 

1. Not Present or Realized Minimal evidence that assigned readings address the disciplinary 


content in science or social studies and give students the opportunity to 
gather information needed to complete the task. Readings are poorly 
aligned to the topic and teaching task, and do not provide students with 
well-balanced perspectives. 
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Dimension 4: Text Appropriateness 


Definition: Degree to which teaching task includes reading texts that are accessible to most students (i.e., 
appropriate for the grade-level and subject matter). 


Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 

- Read and evaluate texts (hard copies or online). 

Student Work 

- References in student work. 

Anchor Readings 

- Read for examples of appropriate reading levels for 3g" grade students. 


5. Fully Realized Assigned readings are highly accessible and appropriate for most 
students in 3°” grade social studies or science classrooms. Selection of 
readings addresses the needs of students with a range of literacy skills, 
including students who are above, at, or below grade level, and English 
Language Learners. 


4. 

3. Moderately Present or Realized Assigned readings are mostly accessible and appropriate for the 
majority of students in 3” grade social studies or science classrooms. 
Selection of readings sufficiently addresses the needs of students with a 
range of literacy skills. 

2. 

1. Not Present or Realized Assigned readings are not accessible or appropriate for students in id 


grade social studies or science classrooms. Selection of readings poorly 
addresses the needs of students with a range of literacy skills. 


Dimension 5: Text Rigor 


Definition: Degree to which teaching task includes reading texts that use and develop academic understanding and 
vocabulary, and offer opportunities for multiple interpretations and higher-order thinking. 


Main Sources of Information: 

Module Creator Handout (Task, Resources, Links) 
- Identify list of selected articles/links. 

- Read and evaluate texts (hard copies or online). 
- Consider issues of source credibility. 

Student Work 

- References in student work. 


5. Fully Realized Assigned readings require students to engage in higher-order thinking, 
and develop a strong academic understanding and vocabulary in social 
studies or science. Readings afford a deep conceptual and contextual 
understanding of the teaching task and topic. Selection of readings 
includes a broad range of credible primary and secondary sources. 


4. 

3. Moderately Present or Realized Assigned readings require students to engage in some higher-order 
thinking, and develop an adequate academic understanding and 
vocabulary in social studies or science. Readings afford a sufficient 
conceptual and contextual understanding of the teaching task and 
topic. Selection of readings includes a moderate range of credible 
primary and secondary sources. 

2. 

1. Not Present or Realized Assigned readings require students to engage in little higher-order 


thinking, or develop an academic understanding and vocabulary in 
social studies or science. Readings afford a limited conceptual and 
contextual understanding of the teaching task and topic. Selection of 
readings ineaes few credible primary and secondary sources. 


Dimension 6: Fidelity to LDC Module Instruction 


Definition: Degree to which module instruction, activities, and teaching task address each of the four stages of 
instructional practice (preparation for the task, reading process, transition to writing, writing process). 


Main Sources of Information: 

Module Creator Handout (Instruction) 

Information Sheet 

- Evaluate for distribution of activities and time spent on each of the four stages of instructional practice. 


5. Fully Realized The module instruction, activities, and teaching task reflect deliberate 
attention and fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect demonstrable effort to develop 
instructional scaffolding within and across each stage of instruction. 


4. 

3. Moderately Present or Realized The module instruction, activities, and teaching task reflect moderate 
attention and fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect sufficient effort to develop 
instructional scaffolding within and across each stage of instruction. 

2. 

1. Not Present or Realized The module instruction, activities, and teaching task reflect poor 


attention and lack of fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect inadequate effort to develop 
instructional scaffolding within and across each stage of instruction. 


Dimension 7: Quality Instructional Strategies 


Definition: Degree to which the module provides clear instructional strategies aimed at helping students develop 
literacy skills and successfully complete the teaching task. And the degree to which module instruction and 
activities scaffold critical thinking and performance in a way that is meaningful within the context of a given field or 
subject-matter. 


Main Sources of Information: 

Module Creator Handout (Instruction) 

Classroom Handouts 

Student Work 

- Evaluate extent to which instructional strategies guide student learning in literacy and ability to complete the 
teaching task. 

- Evaluate extent to which the module activities scaffold critical thinking and student performance within the 
context of the subject matter at the core of the teaching task. 


5. Fully Realized Module provides clear and targeted instructional strategies and 
activities that scaffold student learning and promote critical thinking in 
social studies or science. There is explicit attention to helping students 
develop an accurate understanding of the topic and teaching task, and 
literacy skills necessary to successfully complete the writing task. 


4. 

3. Moderately Present or Realized Instructional strategies and activities are available to support adequate 
student learning and critical thinking in social studies or science. There 
is moderate attention to helping students develop an understanding of 
the topic and teaching task, and literacy skills necessary to complete 
the writing task. 

2. 

1. Not Present or Realized Limited instructional strategies and activities available to support 


student learning and critical thinking in social studies or science. 
Insufficient attention to helping students develop an understanding of 
the topic and teaching task, or literacy skills necessary to complete the 
writing task. 
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Dimension 8: Coherence and Clarity of Module 


Definition: Degree to which there is logical alignment between the teaching task and other module goals with 
readings, mini-tasks, and instructional strategies. 


Main Sources of Information: 
Module Creator Handout 
Classroom Handouts 
Student Work 


5. Fully Realized Strong alignment between the teaching task and goals of the module, 
including the CCSS and local and state literacy and content standards, 
with the readings, mini-tasks, student work, and instructional 


strategies. 

4. 

3. Moderately Present or Realized Moderate alignment between the teaching task and goals of the 
module, including the CCSS and local and state literacy and content 
standards, with the readings, mini-tasks, student work, and 
instructional strategies. 

2. 

1. Not Present or Realized Poor alignment between the teaching task and goals of the module, 


including the CCSS and local and state literacy and content standards, 
with the readings, mini-tasks, student work, and instructional 
strategies. 


Dimension 9: Overall Impression 


Definition: Holistic assessment of LDC Module. 


Main Sources of Information: 
Module Creator Handout 
Classroom Handouts 
Student Work 


Main question: To what extent does this module contribute to student college readiness and development of 
advanced literacy skills? 


5. Advanced LDC Module Implementation 
4. Proficient LDC Module Implementation 
3. Adequate LDC Module Implementation 
2. Marginal LDC Module Implementation 


1. Inadequate LDC Module Implementation 
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Appendix B: Hillsborough Modules 


UCLA CRESST’s study is also evaluating district-wide implementation of LDC in 6% grade 
Advanced Reading classrooms in Hillsborough, FL. The implementation differed significantly from 
what occurred in IU13 and Kentucky for two reasons. First, the modules were designed by literacy 
experts at the district level, not by individual classroom teachers. Second, although the modules 
included template tasks that asked students to write about content (e.g., physical fitness, child 
labor), developers were not saddled with the additional burden of integrating subject-specific 
content standards. Nonetheless, comparison between subjects and between states might reveal 
whether these differences in implementation were evident in module quality as measured by the 
CRESST Assignment Measure. 

We recruited an experienced middle school teacher, certified in social studies and English 
Language Arts, to rate the Hillsborough modules using the CRESST Assignment Measure. In 
comparing the ratings, it is important to keep in mind two limitations: (1) because the Hillsborough 
modules were designed by district leaders and distributed to teachers as mandated curriculum, we 
were effectively in possession of all the instructional materials that constituted instruction ofa 
particular module. Such detailed evidence of module implementation stands in contrast to samples 
we collected from IU13 and Kentucky; (2) whereas a few of the modules include models of student 
work, these are not marked high, medium, and low, and in any event, do not reflect an individual 


teacher’s assessment of student achievement. 


Table 1A: Average Module Ratings by Subject Area 


KY and IU- | Hillsborough: 
KY and IU- 13: 8" 6" grade 
13: 8" grade grade Advanced 
Dimensions Social Studies Science Reading 
* 
1. Effective Writing Task 3.4 3.05 3.75 
2. Alignment to Standards 2.42 3.25 3.13 
* 
3. Text Alignment 3.44 3.52 3.88 
4. Text Appropriateness 3.24 3.41 3.38 
5. Text Rigor 3.35 3.05 3.25 
* 
6. Fidelity to LDC 3.04 3.85 4.00 
* 
7. Quality Instructional Strategies 2.85 3.09 3.75 
* 
8. Coherence and Clarity of Module 2.82 3.23 3.75 
* 
8. Overall Impression 2.73 3.04 3.25 


“Indicates higher means. 


Zo 


Table 1B: Average Module Ratings by State 


Dimensions 1U13 KY FL 
* 
L. Effective Writing Task 3.09 3.33 3.75 
* 
2. Alignment to Standards 2.95 2.66 3.13 
* 
3. Text Alignment 3.38 3.55 3.88 
4. Text Appropriateness 3.09 3.53 3.38 
5. Text Rigor 2.91 3.42 3.25 
* 
6. Fidelity to LDC 3.47 3.35 4.00 
* 
7. Quality Instructional Strategies 2.86 3.03 3.75 
* 
8. Coherence and Clarity of Module 2.82 3.23 3.75 
* 
8. Overall Impression 2.73 3.04 3.25 


“Indicates higher means. 


We see in Tables 1A and 1B that Hillsborough modules scored higher on almost all 
dimensions when compared to the modules collected from 1U13 and KY. This finding should not be 
surprising. The modules were developed by literacy experts and included elaborate lesson plans 
with carefully scaffolded instruction in reading and writing. However, on three dimensions, the 
modules from FL were more or less the same as the others. Putting aside the score on Alignment to 
Standards (D2) in Table 1A, which was difficult to assess since the Hillsborough Modules were not 
required to address content standards, we see that the means for D4 and D5 were roughly the same 
across subject area and across states. This finding may suggest two distinct, but related, challenges 
about LDC implementation in ELA classrooms. First, in choosing topics that appear to have high 
potential for student engagement, developers may struggle to find “naturally occurring” texts that 
are appropriate and aligned with students’ reading level (e.g., one module included dense articles 
produced by UNESCO on global child labor practices). Second, and relatedly, in an effort to find 
readable texts, developers may sacrifice substantive, academic rigor (e.g., many of the readings 
could be characterized as fluff articles about how to build an exercise routine or the dangers of 
texting). 

Despite the methodological limitations of the above comparison, it has crucial policy 
implications for any scale-up of LDC implementation. In particular, we see that module quality may 
be improved if materials are designed and disseminated by instructional experts at the district level. 
On the other hand, the challenge of selecting subject-specific texts that are both age-level 
appropriate and academically rigorous may persist even under such conditions. A more conclusive 
study might compare modules in the same subject area designed at the district level and by 


individual teachers, as a way of investigating the affordances of each approach. 
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Appendix C: 


Descriptive Analyses of Kentucky and Pennsylvania LDC Teacher Surveys 


Table Cl 


Years of Teaching Experience 


Type of experience n Mean Std Dev Minimum Maximum 

Kentucky Science 

Years of teaching 12 11.75 8.25 4 32 

Years taught in current school 12 6.67 3.09 3 13 

Years taught in current district 12 7:23 3.70 3 16 
Kentucky Social Studies 

Years of teaching 14 12.00 8.19 3 33 

Years taught in current school 14 8.93 views 2 30 

Years taught in current district 14 9.71 8.17 2 30 
Pennsylvania Science 

Years of teaching 7 13.71 11.94 3 36 

Years taught in current school 7 12.14 12.46 2 35 

Years taught in current district a 2257 12.10 3 35 
Pennsylvania Social Studies 

Years of teaching 7 8.00 3.70 3 14 

Years taught in current school 7 6.43 3.51 2 12 

Years taught in current district 7 6.93 3.27 3 12 


Table C2 


Teaching of Different Student Populations 


Student population n Yes (%) n No (%) 
Kentucky Science (7 = 12) 
ELL students 7 58.3 5 41.7 
Special education students 11 91.7 1 8.3 
Students reading or writing below grade level 11 91.7 1 8.3 
Students with advanced literacy levels 12 100.0 0 0.0 
Kentucky Social Studies (n = 14) 
ELL students 6 42.9 8 57.1 
Special education students 12 85.7 2 14.3 
Students reading or writing below grade level 13 92.9 1 7.1 
Students with advanced literacy levels 14 100.0 0 0.0 
Pennsylvania Science (n = 7) 
ELL students 5 71.4 2 28.6 
Special education students 7 100.0 0 0.0 
Students reading or writing below grade level 6 85.7 1 14.3 
Students with advanced literacy levels 7 100.0 0 0.0 
Pennsylvania Social Studies (” = 7) 
ELL students 6 85.7 1 14.3 
Special education students 6 85.7 1 14.3 
Students reading or writing below grade level 7 100.0 0 0.0 
Students with advanced literacy levels 7 100.0 0 0.0 
Table C3 
Type of Participation in the LDC Initiative 
Required Voluntary 
State and Subject n % n % 
Kentucky Science 11 91.7 1 7.7 
Kentucky Social Studies 9 64.3 5 35.7 
Pennsylvania Science 3 50.0 3 50.0 
Pennsylvania Social Studies 5 71.4 2 28.6 


Table C4 
Modules Developed and Taught 


School year n Mean Std Dev Minimum Maximum 


Kentucky Science 


Modules taught in 2011-12 12 0.83 0.835 0 2 

Modules developed in 2012-13 12 1.92 0.793 1 4 

Modules taught in 2012-13 12 1.58 0.515 1 2 
Kentucky Social Studies 

Modules taught in 2011-12 14 1.93 0.997 0 4 

Modules developed in 2012-13 13 1.69 0.630 1 3 

Modules taught in 2012-13 14 2.36 0.745 2 4 
Pennsylvania Science 

Modules taught in 2011-12 6 1.33 0.516 1 2 

Modules developed in 2012-13 ) 1.80 0.837 1 3 

Modules taught in 2012-13 6 2.67 1.862 1 6 
Pennsylvania Social Studies 

Modules taught in 2011-12 7 1.57 0.535 1 2 

Modules developed in 2012-13 7 1.86 0.378 1 2 

Modules taught in 2012-13 7 2.71 0.756 1 3 


Table C5 


Teacher Attitudes Regarding Literacy Instruction in Content Area Classrooms 


Question n Mean Std Dev 


Kentucky Science 


Teachers from all content areas should help students improve 


their reading and writing skills. va aa ad 
Science and social studies teachers do not have time to teach 2 L.17 0.94 
reading and writing. 
Writing assignments can help my students develop a deeper 
; i 12 2.67 0.49 
understanding of important concepts. 
Kentucky Social Studies 
Teachers from all content areas should help students improve 
: ; me : 14 2.71 0.47 
their reading and writing skills. 
Science and social studies teachers do not have time to teach 14 1.00 0.96 
reading and writing. 
Writing assignments can help my students develop a deeper 
f j 14 2.71 0.47 
understanding of important concepts. 
Pennsylvania Science 
Teachers from all content areas should help students improve 
. ; oe : 6 2.83 0.41 
their reading and writing skills. 
Science and social studies teachers do not have time to teach 6 0.50 0.55 
reading and writing. 
Writing assignments can help my students develop a deeper 
: ‘ 6 2.83 0.41 
understanding of important concepts. 
Pennsylvania Social Studies 
Teachers from all content areas should help students improve 
: ; i ‘ 7 3.00 0.00 
their reading and writing skills. 
Science and social studies teachers do not have time to teach 7 0.57 0.79 
reading and writing. 
Writing assignments can help my students develop a deeper 7 71 0.49 


understanding of important concepts. 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C6 


Percent of Class Time Spent on the Different LDC Components 


Components n Mean Std Dev Minimum Maximum 

Kentucky Science 

Prepare for task/introduce module 12 26.67 21.36 10 80 

Reading process 12 22.50 8.66 10 40 

Transition to writing 12 15.83 9.00 0 30 

Writing process 12 35.00 15.08 10 60 
Kentucky Social Studies 

Prepare for task/introduce module 14 17.86 14.24 5 60 

Reading process 14 24.29 9.78 10 45 

Transition to writing 14 20.71 8.52 10 35 

Writing process 14 37.14 12.20 15 60 
Pennsylvania Science 

Prepare for task/introduce module 6 17.50 2.74 15 20 

Reading process 6 29.17 12.42 15 45 

Transition to writing 6 20.83 5.85 15 30 

Writing process 6 32.50 12.15 15 50 
Pennsylvania Social Studies 

Prepare for task/introduce module 7 12.86 6.36 5 25 

Reading process 7 32.14 9.51 15 40 

Transition to writing 7 19.29 8.86 10 30 

Writing process 7 35.71 6.73 25 45 


Note. Means represent percents. 


Table C7 


Degree of Emphasis Placed on Different Reading Strategies during LDC Instruction (Kentucky Science) 


Strategies n No Little Some A great deal 
emphasis emphasis emphasis of emphasis 
(%) (%) (%) (%) 

Independent reading/ research 12 0.0 8.3 41.7 50.0 
Making predictions/ previewing 12 8.3 25.0 50.0 16.7 
Summarizing important points 12 0.0 0.0 50.0 50.0 
Note-taking/ annotation 12 0.0 0.0 50.0 50.0 
Identifying/ defining vocabulary 12 0.0 8.3 66.7 25.0 
Analyzing text structure 12 16.7 25.0 41.7 16.7 
eee information from graphical 2 83 83 50.0 33.3 
Distinguishing fact from opinion 12 0.0 16.7 33.3 50.0 
Drawing conclusions from textual evidence 12 0.0 0.0 41.7 58.3 
Citing textual evidence to support claims 12 0.0 16.7 33.3 50.0 
Evaluating strength/ weakness of evidence 12 8.3 16.7 58.3 16.7 
Comparing arguments in two or more texts 12 0.0 33.3 25.0 41.7 
Examining author’s perspective/ bias 12 25.0 50.0 8.3 16.7 
Examining rhetorical devices 12 50.0 50.0 0.0 0.0 


Table C8 


Degree of Emphasis Placed on Different Reading Strategies during LDC Instruction (Kentucky Social Studies) 


Strategies n No Little Some A great deal 

emphasis emphasis emphasis of emphasis 
(%) (%) (%) (%) 
Independent reading/ research 14 7.1 7.1 S71 28.6 
Making predictions/ previewing 14 7.1 14.3 64.3 14.3 
Summarizing important points 14 0.0 tA 28.6 64.3 
Note-taking/ annotation 14 0.0 14.3 42.9 42.9 
Identifying/ defining vocabulary 14 0.0 7.1 42.9 50.0 
Analyzing text structure 14 0.0 28.6 35.7 35.7 
Interpreting information from graphical 14 0.0 14.3 64.3 21.4 

text 

Distinguishing fact from opinion 14 71 14.3 28.6 50.0 
Drawing conclusions from textual evidence 14 0.0 0.0 21.4 78.6 
Citing textual evidence to support claims 14 0.0 0.0 35.7 64.3 
Evaluating strength/ weakness of evidence 14 0.0 21.4 42.9 35.7 
Comparing arguments in two or more texts 14 0.0 21.4 21.4 57.1 
Examining author’s perspective/ bias 14 0.0 21.4 50.0 28.6 
Examining rhetorical devices 14 21.4 28.6 35.7 14.3 


Table C9 


Degree of Emphasis Placed on Different Reading Strategies during LDC Instruction (Pennsylvania Science) 


Strategies No Little Some A great deal 

emphasis emphasis emphasis of emphasis 
(%) (%) (%) (%) 
Independent reading/ research 6 0.0 0.0 16.7 83.3 
Making predictions/ previewing 6 16.7 33.3 16.7 33.3 
Summarizing important points 6 0.0 0.0 16.7 83.3 
Note-taking/ annotation 6 0.0 0.0 16.7 83.3 
Identifying/ defining vocabulary 6 16.7 0.0 50.0 33.3 
Analyzing text structure 6 33.3 16.7 16.7 33.3 
Interpreting information from graphical 6 16.7 33.3 33.3 16.7 

text 
Distinguishing fact from opinion 6 16.7 0.0 83.3 0.0 
Drawing conclusions from textual 6 0.0 0.0 50.0 50.0 
evidence 

Citing textual evidence to support claims 6 16.7 0.0 33.3 50.0 
Evaluating strength/ weakness of evidence 6 16.7 16.7 66.7 0.0 
Comparing arguments in two or more texts 6 16.7 16.7 33.3 33.3 
Examining author’s perspective/ bias 6 16.7 33.3 33.3 16.7 
Examining rhetorical devices 6 66.7 16.7 16.7 0.0 


Table C10 


Degree of Emphasis Placed on Different Reading Strategies during LDC Instruction (Pennsylvania Social 


Studies) 
Strategies No Little Some A great deal 
emphasis emphasis emphasis of emphasis 
(%) (%) (%) (%) 
Independent reading/ research 7 0.0 0.0 28.6 71.4 
Making predictions/ previewing 7 0.0 14.3 85.7 0.0 
Summarizing important points 7 0.0 0.0. 0.0 100.0 
Note-taking/ annotation 7 0.0 0.0 28.6 71.4 
Identifying/ defining vocabulary ip 0.0 0.0 57.1 42.9 
Analyzing text structure 7 0.0 42.9 42.9 14.3 
Interpreting information from graphical 7 0.0 28.6 57.1 14.3 
text 
Distinguishing fact from opinion 7 14.3 14.3 42.9 28.6 
Drawing conclusions from textual 7 0.0 14.3 28.6 57.1 
evidence 
Citing textual evidence to support claims i 0.0 28.6 28.6 42.9 
Evaluating strength/ weakness of evidence y/ 0.0 28.6 57.1 14.3 
Comparing arguments in two or more texts 7 0.0 28.6 57.1 14.3 
Examining author’s perspective/ bias 7 0.0 14.3 71.4 14.3 
Examining rhetorical devices 7 14.3 71.4 14.3 0.0 


Table C11 


Degree of Emphasis Placed on Different Writing Strategies during LDC Instruction (Kentucky Science) 


Strategies n No Little Some A great deal 

emphasis emphasis emphasis of emphasis 
(%) (%) (%) (%) 
Generating ideas for writing 12 0.0 8.3 50.0 41.7 
Outlining 12 0.0 8.3 41.7 50.0 
Writing/ text structure 12 0.0 8.3 8.3 83.3 
Formulating a thesis statement 12 0.0. 0.0 25.0 75.0 
Formulating a counter-argument 12 8.3 8.3 33.3 50.0 
Writing an introduction 12 0.0 0.0. 25.0 75.0 
Writing a conclusion 12 0.0 8.3 33.3 58.3 
Writing a body paragraph 12 0.0 0.0 16.7 83.3 
Using transitional words or phrases 12 0.0 33.3 25.0 41.7 
Incorporating quotes/ evidence 12 0.0 8.3 25.0 66.7 


Table C12 


Degree of Emphasis Placed on Different Writing Strategies during LDC Instruction (Kentucky Social Studies) 


Strategies n No Little Some A great deal 

emphasis emphasis emphasis of emphasis 
(%) (%) (%) (%) 
Generating ideas for writing 14 21.4 28.6 35.7 14.3 
Outlining 14 0.0 14.3 35.7 50.0 
Writing/ text structure 14 0.0 7.1 50.0 42.9 
Formulating a thesis statement 14 0.0 0.0 35.7 64.3 
Formulating a counter-argument 14 0.0 0.0 28.6 71.4 
Writing an introduction 14 0.0 14.3 42.9 42.9 
Writing a conclusion 14 0.0 0.0 21.4 78.6 
Writing a body paragraph 14 0.0 7.1 28.6 64.3 
Using transitional words or phrases 14 0.0 0.0 14.3 85.7 
Incorporating quotes/ evidence 14 0.0 21.4 42.9 35.7 
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Table C13 


Degree of Emphasis Placed on Different Writing Strategies during LDC Instruction (Pennsylvania Science) 


Strategies n No Little Some A great deal 
emphasis emphasis emphasis of emphasis 
(%) (%) (%) (%) 

Generating ideas for writing 6 16.7 16.7 16.7 50.0 
Outlining 6 0.0 0.0 50.0 50.0 
Writing/ text structure 6 0.0 16.7 33.3 50.0 
Formulating a thesis statement 6 16.7 16.7 50.0 16.7 
Formulating a counter-argument 6 50.0 16.7 16.7 16.7 
Writing an introduction 6 0.0 16.7 33.3 50.0 
Writing a conclusion 6 0.0 16.7 33.3 50.0 
Writing a body paragraph 6 0.0 16.7 33.3 50.0 
Using transitional words or phrases 6 33.3 0.0 50.0 16.7 
Incorporating quotes/ evidence 6 16.7 0.0 16.7 66.7 


Table C14 
Degree of Emphasis Placed on Different Writing Strategies during LDC Instruction (Pennsylvania Social 
Studies) 

Strategies n No Little Some A great deal 

emphasis emphasis emphasis of emphasis 
(%) (%) (%) (%) 

Generating ideas for writing 7 0.0 0.0 71.4 28.6 
Outlining m 0.0 0.0 57.1 42.9 
Writing/ text structure 7 0.0 0.0 42.9 57.1 
Formulating a thesis statement 7 0.0 0.0 14.3 85.7 
Formulating a counter-argument 7 14.3 0.0 71.4 14.3 
Writing an introduction qi) 0.0 0.0 28.6 71.4 
Writing a conclusion ai 0.0 0.0 14.3 85.7 
Writing a body paragraph 7 0.0 0.0 14.3 85.7 
Using transitional words or phrases 7 14.3 0.0 85.7 0.0 
Incorporating quotes/ evidence i 0.0 14.3 42.9 42.9 
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Table C15 


Frequency of Use of Strategies to Assess Student Understanding (Kentucky Science) 


Strategies n Never (%) Rarely (%) Sometimes Often (%) 
(%) 
Listened as students discussed reading or 2 0.0 0.0 50.0 50.0 
writing with peers 
Asked students to provide feedback to 2 0.0 83 50.0 417 
each other 
Circulated and reviewed student notes and 2 0.0 0.0 16.7 833 
work 
Collected and reviewed student writing 2 0.0 0.0 583 417 
exercises 
Asked students to answer oral questions 12 0.0 16.7 25.0 58.3 
Reviewed student rough drafts 12 0.0 0.0 33.3 66.7 
Asked certain students to present writing 2 417 333 250 0.0 
to class 
Assigned a quiz 12 33.3 41.7 25.0 0.0 
Graded student work 12 0.0 8.3 16.7 75.0 
Exit slips 12 33.3 16.7 33.3 16.7 
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Table C16 


Frequency of Use of Strategies to Assess Student Understanding (Kentucky Social Studies) 


Strategies Never (%) Rarely (%) Sometimes Often (%) 
(%) 
Listened as students discussed reading or 14 0.0 14 57.1 14 
writing with peers 
Asked students to provide feedback to 14 0.0 35.7 42.9 14 
each other 
Circulated and reviewed student notes and 14 0.0 0.0 42.9 57.1 
work 
Collected and reviewed student writing 14 0.0 0.0 42.9 57.1 
exercises 
Asked students to answer oral questions 14 0.0 0.0 35.7 64.3 
Reviewed student rough drafts 14 0.0 0.0 28.6 71.4 
Asked certain students to present writing 14 1.4 42.9 86 71 
to class 
Assigned a quiz 14 28.6 57.1 14.3 0.0 
Graded student work 14 0.0 7A 28.6 64.3 
Exit slips 14 0.0 21.4 35.7 42.9 
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Table C17 


Frequency of Use of Strategies to Assess Student Understanding (Pennsylvania Science) 


Strategies Never (%) Rarely (%) Sometimes Often (%) 
(%) 
Listened as students discussed reading or 0.0 0.0. 16.7 83.3 
writing with peers 
Asked students to provide feedback to 0.0 0.0. 33.3 66.7 
each other 
Circulated and reviewed student notes and 0.0 16.7 0.0 833 
work 
Collected and reviewed student writing 0.0 16.7 833 0.0 
exercises 
Asked students to answer oral questions 16.7 0.0 33.3 50.0 
Reviewed student rough drafts 16.7 0.0 50.0 33.3 
Asked certain students to present writing 333 50.0 0.0 16.7 
to class 
Assigned a quiz 100.0 0.0 0.0 0.0 
Graded student work 0.0 16.7 16.7 66.7 
Exit slips 50.0 50.0 0.0. 0.0 
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Table C18 


Frequency of Use of Strategies to Assess Student Understanding (Pennsylvania Social Studies) 


Strategies Never (%) Rarely (%) Sometimes Often (%) 
(%) 
Listened as students discussed reading or 0.0 0.0 143 85.7 
writing with peers 
Asked students to provide feedback to 0.0 0.0 86 71.4 
each other 
Circulated and reviewed student notes and 0.0 0.0 28.6 14 
work 
Collected and reviewed student writing 0.0 143 42.9 42.9 
exercises 
Asked students to answer oral questions 0.0 0.0 28.6 71.4 
Reviewed student rough drafts 0.0 0.0 42.9 57.1 
Asked certain students to present writing 86 143 42.9 143 
to class 
Assigned a quiz 57.1 14.3 28.6 0.0 
Graded student work 0.0 0.0 14.3 85.7 
Exit slips 0.0 42.9 57.1 0.0 
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Table C19 


Frequency of Use of Strategies to Respond to Misunderstanding (Kentucky Science) 


Strategies n Never (%) Rarely (%) Sometimes Often (%) 
(%) 

Held one-on-one conference with student 12 8.3 16.7 25.0 50.0 
Asked peer to provide feedback or 2 8.3 16.7 41.7 33.3 
organized peer editing session 

Stopped class and modeled strategy 12 0.0 25.0 50.0 25.0 
Scheduled in-class workshop time 12 8.3 8.3 33.3 50.0 
Wrote specific comments on student work 12 0.0 8.3 25.0 66.7 
Offered student a hint or suggestion 12 0.0. 8.3 33.3 58.3 
Gave student more time to try again and 2 0.0 0.0 41.7 58.3 
self-correct 

Gave student the answer 12 16.7 58.3 25.0 0.0 
Graded student work 12 0.0 8.3 25.0 66.7 
Re-taught lesson segment 12 0.0 16.7 75.0 8.3 
Planned to review skill in later lessons 12 0.0 50.0 50.0 0.0 
Assigned grammar exercises 12 91.7 8.3 0.0 0.0 
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Table C20 


Frequency of Use of Strategies to Respond to Misunderstanding (Kentucky Social Studies) 


Strategies n Never (%) Rarely (%) Sometimes Often (%) 
(%) 
Held one-on-one conference with student 14 0.0 7.1 35.7 57.1 
a ee Ce 
Stopped class and modeled strategy 14 0.0 7.1 85.7 7.1 
Scheduled in-class workshop time 14 0.0 71 28.6 64.3 
Wrote specific comments on student work 14 0.0 0.0 14.3 85.7 
Offered student a hint or suggestion 14 0.0 iP 35.7 57.1 
pane ae more time to try again and 14 0.0 0.0 14 718.6 
Gave student the answer 14 35.7 50.0 14.3 0.0 
Graded student work 14 0.0 7A 28.6 64.3 
Re-taught lesson segment 14 7.1 21.4 71.4 0.0 
Planned to review skill in later lessons 14 7.1 7.1 64.3 21.4 
Assigned grammar exercises 14 28.6 64.3 7.1 0.0 
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Table C21 


Frequency of Use of Strategies to Respond to Misunderstanding (Pennsylvania Science) 


Strategies Never (%) Rarely (%) Sometimes Often (%) 
(%) 
Held one-on-one conference with student 6 16.7 0.0 16.7 66.7 
cr oe tes ee rr 
Stopped class and modeled strategy 6 16.7 0.0 66.7 16.7 
Scheduled in-class workshop time 6 33.3 0.0 33.3 33.3 
Wrote specific comments on student work 6 16.7 16.7 33.3 33.3 
Offered student a hint or suggestion 6 16.7 0.0 50.0 33.3 
pane ae more time to try again and 6 16.7 0.0 50.0 33.3 
Gave student the answer 6 50.0 33.3 16.7 0.0 
Graded student work 6 0.0 0.0 33.3 66.7 
Re-taught lesson segment 6 16.7 0.0 83.3 0.0 
Planned to review skill in later lessons 6 16.7 0.0 83.3 0.0 
Assigned grammar exercises 6 100.0 0.0 0.0 0.0 
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Table C22 


Frequency of Use of Strategies to Respond to Misunderstanding (Pennsylvania Social Studies) 


Strategies Never (%)  Rarely(%) Sometimes Often (%) 
(%) 
Held one-on-one conference with student 7 0.0 14.3 14.3 71.4 
miedo persuneecsign © i 0 " a we 
Stopped class and modeled strategy 7 0.0 0.0 57.1 42.9 
Scheduled in-class workshop time 7 0.0 14.3 14.3 71.4 
Wrote specific comments on student work vd 0.0 0.0 28.6 71.4 
Offered student a hint or suggestion 7 0.0 0.0 28.6 71.4 
pane ae more time to try again and 7 0.0 0.0 42.9 57.1 
Gave student the answer 7 14.3 85.7 0.0 0.0 
Graded student work 7 0.0 0.0 42.9 57.1 
Re-taught lesson segment 7 0.0 42.9 57.1 0.0 
Planned to review skill in later lessons 7 0.0 14.3 57.1 28.6 
Assigned grammar exercises 7 85.7 14.3 0.0 0.0 
Table C23 
School and District Support for LDC (Kentucky Science) 
Question n Mean Std Dev 

School administrators ... 

Have a firm understanding of the LDC framework 9 2.00 1.00 

Have made formative assessment a priority at my school 12 2.58 0.52 

Encouraged me to participate in the LDC initiative 12 2.17 0.84 

a a with feedback about my instruction of the D 0.75 1.14 

Expressed concerns that teaching modules is taking time away 

from other instructional priorities S 096 ae 

pier eae the LDC framework is aligned with other 10 1.50 0.97 
District administrators ... 

Support the LDC framework 12 2.50 0.52 
Have a firm understanding of the LDC framework 10 1.80 1.23 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C24 
School and District Support for LDC (Kentucky Social Studies) 


Question N Mean Std Dev 
School administrators ... 
Have a firm understanding of the LDC framework 13 2.31 1.11 
Have made formative assessment a priority at my school 14 2.36 0.75 
Encouraged me to participate in the LDC initiative 14 2.86 0.36 
Provided me with feedback about my instruction of the 14 1.43 1.09 
module(s) 
Expressed concerns that teaching modules is taking time away 
: : zee 13 0.54 0.88 
from other instructional priorities 
Communicated how the LDC framework is aligned with other 
aciae 14 2.07 1.21 
school initiatives 
District administrators ... 
Support the LDC framework 14 2.93 0.27 
Have a firm understanding of the LDC framework 14 2.71 0.61 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C25 
School and District Support for LDC (Pennsylvania Science) 


Question n Mean Std Dev 
School administrators ... 
Have a firm understanding of the LDC framework 5 2.40 0.55 
Have made formative assessment a priority at my school ) 2.00 1.41 
Encouraged me to participate in the LDC initiative 6 3.00 0.00 
Provided me with feedback about my instruction of the 
5 1.00 1.41 
module(s) 
Expressed concerns that teaching modules is taking time away 
; : Seen 5 0.40 0.55 
from other instructional priorities 
Communicated how the LDC framework is aligned with other 
marae 5 1.60 1.14 
school initiatives 
District administrators ... 
Support the LDC framework 5 2.60 0.89 
Have a firm understanding of the LDC framework 5 2.80 0.45 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C26 
School and District Support for LDC (Pennsylvania Social Studies) 


Question n Mean Std Dev 
School administrators ... 
Have a firm understanding of the LDC framework 7 2.57 0.54 
Have made formative assessment a priority at my school 7 2.29 0.76 
Encouraged me to participate in the LDC initiative 7 2.57 0.79 
Provided me with feedback about my instruction of the 7 1.86 1.07 
module(s) 
Expressed concerns that teaching modules is taking time away 6 LW 1.33 
from other instructional priorities ‘ 
Communicated how the LDC framework is aligned with other 
Rancioe 6 2.00 0.63 
school initiatives 
District administrators ... 
Support the LDC framework 6 2.33 0.82 
Have a firm understanding of the LDC framework 6 1.83 1.33 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C27 


Individuals who Visited Teachers’ Classrooms during LDC Module Instruction 


Individuals n Yes (%) n No (%) 

Kentucky Science (7 = 12) 

District or network LDC project lead 2 16.7 10 83.3 

Principal 4 33.3 8 66.7 

Instructional coach/department head 1 8.3 11 91.7 

Teacher colleague 2 16.7 10 83.3 
Kentucky Social Studies (n = 14) 

District or network LDC project lead 6 42.9 8 57.1 

Principal 7 50.0 7 50.0 

Instructional coach/department head 4 28.6 10 71.4 

Teacher colleague 6 42.9 8 57.1 
Pennsylvania Science (n = 6) 

District or network LDC project lead 2 33.3 4 66.7 

Principal 2 33.3 4 66.7 

Instructional coach/department head 1 16.7 5 83.3 

Teacher colleague y 33.3 4 66.7 
Pennsylvania Social Studies (” = 7) 

District or network LDC project lead 3 42.9 4 57.1 

Principal 6 85.7 1 14.3 

Instructional coach/department head 1 14.3 6 85.7 

Teacher colleague 5 71.4 2 28.6 


Table C28 


Teacher Participation in LDC Professional Development in 2012-13 School Year 


District/Subject Yes (%) n No (%) 
Kentucky Science (n = 12) 50.0 6 50.0 
Kentucky Social Studies (n = 14) 85.7 2 14.3 
Pennsylvania Science (n = 6) 50.0 3 50.0 
Pennsylvania Social Studies (” = 7) 85.7 1 14.3 


Ze 


Table C29 


Number of Professional Development Sessions Attended in 2012-13 School Year 


District/Subject n Mean Std Dev Minimum Maximum 
Kentucky Science 6 3.17 1.84 1.00 6.00 
Kentucky Social Studies 12 2.83 0.58 2.00 4.00 
Pennsylvania Science 3 2.33 0.58 2.00 3.00 
Pennsylvania Social Studies 6 2.67 1.37 1.00 5.00 


Table C30 


Types and Perceived Effectiveness of Professional Development in which Teachers Participated (n = 6) 


(Kentucky Science) 


Participated If participated, perceived effectiveness 

Type n Yes (%) n Yes (%) n No (%) 
One-on-one classroom visits 0 0.0 0 0.0 0 0.0 
Coaching 4 66.7 0 0.0 0 0.0 
Webinars 0 0.0 0 0.0 0 0.0 
Small group meetings 4 66.7 0 0.0 0 0.0 
School-wide meetings 4 66.7 0 0.0 0 0.0 
District wide-meetings 4 66.7 0 0.0 0 0.0 
Cross-district meetings 0 0.0 0 0.0 0 0.0 


Note. Only teachers who indicated that they participated in a specific type of PD were asked about the 


effectiveness. 
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Table C31 


Types and Perceived Effectiveness of Professional Development in which Teachers Participated (n = 12) 
(Kentucky Social Studies) 


Participated If participated, perceived effectiveness 

Type n Yes (%) n Yes (%) n No (%) 
One-on-one classroom visits 3 25.0 3 100.0 0 0.0 
Coaching 5 41.7 1 100.0 0 0.0 
Webinars 0 0.0 0 0.0 0 0.0 
Small group meetings 11 91.7 3 100.0 0 0.0 
School-wide meetings 7 58.3 3 100.0 0 0.0 
District wide-meetings 9 75.0 3 100.0 0 0.0 
Cross-district meetings 1 8.3 0 0.0 0 0.0 


Note. Only teachers who indicated that they participated in a specific type of PD were asked about the 
effectiveness. 


Table C32 


Types and Perceived Effectiveness of Professional Development in which Teachers Participated (n = 3) 
(Pennsylvania Science) 


Participated If participated, perceived effectiveness 

Type n Yes (%) n Yes (%) n No (%) 
One-on-one classroom visits 1 33.3 1 100.0 0 0.0 
Coaching 1 33.3 0 0.0 0 0.0 
Webinars 3 100.0 1 100.0 0 0.0 
Small group meetings 2 66.7 1 100.0 0 0.0 
School-wide meetings 0 0.0 0 0.0 0 0.0 
District wide-meetings 0 0.0 0 0.0 0 0.0 
Cross-district meetings 2 66.7 1 100.0 0 0.0 


Note. Only teachers who indicated that they participated in a specific type of PD were asked about the 
effectiveness. 
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Table C33 


Types and Perceived Effectiveness of Professional Development in which Teachers Participated (n = 6) 
(Pennsylvania Social Studies) 


Participated If participated, perceived effectiveness 

Type n Yes (%) n Yes (%) n No (%) 
One-on-one classroom visits 3 50.0 3 100.0 0 0.0 
Coaching 4 66.7 2 100.0 0 0.0 
Webinars 4 66.7 1 50.0 1 50.0 
Small group meetings 5 83.3 3 100.0 0 0.0 
School-wide meetings 4 66.7 3 100.0 0 0.0 
District wide-meetings 2 33.3 1 100.0 0 0.0 
Cross-district meetings 4 66.7 3 100.0 0 0.0 


Note. Only teachers who indicated that they participated in a specific type of PD were asked about the 
effectiveness. 


Table C34 


Content Included in LDC Professional Development Sessions in which Teachers Participated (1 = 6) (Kentucky 
Science) 


Question n Yes (%) n No (%) 
Using LDC as a way to implement CCSS 3 50.0 3 50.0 
Building a teaching task 5 83.3 1 16.7 
Finding appropriate content materials 2 33.3 4 66.7 
Designing modules 5 83.3 1 16.7 
Using the instructional ladder 5 83.3 1 16.7 
Using mini-tasks to address reading and writing skills 4 66.7 2 33.3 
Providing students with feedback on their writing 3 50.0 3 50.0 
Scoring student work with LDC rubric 4 66.7 2 33.3 
Building modules with Module Creator 6 100.0 0 0.0 
Differentiating module instruction to meet student needs 2 33.3 4 66.7 
Implementing modules with special education students 0 0.0 6 100.0 
Implementing modules with ELL students 0 0.0 6 100.0 
eee modules with students who read/write below 0 0.0 6 100.0 
Implementing modules with students with advanced literacy 0 0.0 6 100.0 


levels 
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Table C35 


Content Included in LDC Professional Development Sessions in which Teachers Participated ( = 12) 


(Kentucky Social Studies) 


Question n Yes (%) n No (%) 
Using LDC as a way to implement CCSS 10 71.4 2 16.7 
Building a teaching task 12 100.0 0 0.0 
Finding appropriate content materials 11 91.7 1 8.3 
Designing modules 12 100.0 0 0.0 
Using the instructional ladder 10 71.4 2 16.7 
Using mini-tasks to address reading and writing skills 12 100.0 0 0.0 
Providing students with feedback on their writing 6 50.0 6 50.0 
Scoring student work with LDC rubric 9 75.0 3 25.0 
Building modules with Module Creator 11 91.7 1 8.3 
Differentiating module instruction to meet student needs 9 75.0 25.0 
Implementing modules with special education students 4 33.3 8 66.7 
Implementing modules with ELL students 2 16.7 10 71.4 
Se a modules with students who read/write below 5 417 7 583 
Implementing modules with students with advanced literacy 5 417 7 583 


levels 
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Table C36 


Content Included in LDC Professional Development Sessions in which Teachers Participated (n = 3) 
(Pennsylvania Science) 


Question n Yes (%) n No (%) 

Using LDC as a way to implement CCSS 3 100.0 0 0.0 
Building a teaching task 3 100.0 0 0.0 
Finding appropriate content materials 3 100.0 0 0.0 
Designing modules 3 100.0 0 0.0 
Using the instructional ladder 2 66.7 1 33.3 
Using mini-tasks to address reading and writing skills 3 100.0 0 0.0 
Providing students with feedback on their writing 2 66.7 1 33.3 
Scoring student work with LDC rubric 2 66.7 1 33.3 
Building modules with Module Creator 3 100.0 0 0.0 
Differentiating module instruction to meet student needs 3 100.0 0 0.0 
Implementing modules with special education students 3 100.0 0 0.0 
Implementing modules with ELL students 2 66.7 1 33.3 
Implementing modules with students who read/write below 3 100.0 0 0.0 
grade level 

Implementing modules with students with advanced literacy 3 100.0 0 0.0 


levels 


2H 


Table C37 


Content Included in LDC Professional Development Sessions in which Teachers Participated (n = 6) 


(Pennsylvania Social Studies) 


Question Yes (%) n No (%) 
Using LDC as a way to implement CCSS 5 83.3 1 16.7 
Building a teaching task 5 83.3 1 16.7 
Finding appropriate content materials 5 83.3 1 16.7 
Designing modules 6 100.0 0 0.0 
Using the instructional ladder 5 83.3 1 16.7 
Using mini-tasks to address reading and writing skills 5 83.3 1 16.7 
Providing students with feedback on their writing 4 66.7 2 33.3 
Scoring student work with LDC rubric 6 100.0 0 0.0 
Building modules with Module Creator 6 100.0 0 0.0 
Differentiating module instruction to meet student needs 4 66.7 ) 33.3 
Implementing modules with special education students 3 50.0 3 50.0 
Implementing modules with ELL students 2 33.3 4 66.7 
pene modules with students who read/write below 5 833 1 16.7 
Implementing modules with students with advanced literacy A 66.7 2 333 
levels 
Table C38 
Regularly Scheduled Common Planning Time with Colleagues to Discuss LDC 

State/Subject n Yes (%) n No (%) 
Kentucky Science (nm = 12) 7 58.3 5 41.7 
Kentucky Social Studies (n = 14) 12 85.7 2 14.3 
Pennsylvania Science (n = 6) 1 16.7 5 83.3 
Pennsylvania Social Studies (1 = 7) 4 57.1 3 42.9 
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Table C39 


Frequency of Formal and Informal Teacher Collaboration around LDC (Kentucky Science) 


Frequency n Scheduled Meetings (%) n Informal Discussions (%) 
At least once a week 3 42.9 4 33.3 
Every other week 0 0.0 1 8.3 
Once a month 2 28.6 4 33.3 
Once per quarter/trimester/semester 2 28.6 3 25.0 
Never 0 0.0 0 0.0 


Note. Scheduled meetings (n = 7), informal meetings (n = 12) 


Table C40 


Frequency of Formal and Informal Teacher Collaboration around LDC (Kentucky Social Studies) 


Frequency n Scheduled Meetings (%) n Informal Discussions (%) 
At least once a week 2 16.7 4 28.6 
Every other week 2 16.7 3 21.4 
Once a month 2 16.7 4 28.6 
Once per quarter/trimester/semester 6 50.0 3 21.4 
Never 0 0.0 0 0.0 
Note. Scheduled meetings (n = 12), informal meetings (7 = 14) 
Table C41 
Frequency of Formal and Informal Teacher Collaboration around LDC (Pennsylvania Science) 
Frequency n Scheduled Meetings (%) n Informal Discussions (%) 
At least once a week 1 100.0 2 33.3 
Every other week 0 0.0 1 16.7 
Once a month 0 0.0 2 33.3 
Once per quarter/trimester/semester 0 0.0 1 16.7 
Never 0 0.0 0 0.0 


Note. Scheduled meetings (” = 1), informal meetings (” = 6) 
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Table C42 


Frequency of Formal and Informal Teacher Collaboration around LDC (Pennsylvania Social Studies) 


Frequency n Scheduled Meetings (%) n Informal Discussions (%) 
At least once a week 0 0.0 3 42.9 
Every other week 0 0.0 3 42.9 
Once a month 3 75.0 1 14.3 
Once per quarter/trimester/semester 1 25.0 0 0.0 
Never 0 0.0 0 0.0 


Note. Scheduled meetings (7 = 4), informal meetings (” = 7) 


Table C43 


Perceptions of Teacher Collaboration during LDC Implementation (Kentucky Science) 


Question n Mean Std Dev 
I would describe my LDC colleagues as collaborative 12 2.58 0.79 
Collaboration with my LDC colleagues helps me ... 
More effectively use the LDC framework 12 2.17 0.84 
Better support student learning 12 2.00 1.04 
Develop LDC modules 12 2.25 0.75 
Teach LDC modules 12 2.00 0.95 
Revise LDC modules 12 2.08 1.00 
Use the LDC framework rubric 12 1.83 1.12 
Use students’ products to inform my instruction 12 2.00 0.95 


Provide helpful feedback to students about their 
writing 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C44 


Perceptions of Teacher Collaboration during LDC Implementation (Kentucky Social Studies) 


Question n Mean Std Dev 

I would describe my LDC colleagues as collaborative 14 2.71 

Collaboration with my LDC colleagues helps me ... 0.61 
More effectively use the LDC framework 14 2.57 0.65 
Better support student learning 14 2.57 0.65 
Develop LDC modules 14 2.57 0.65 
Teach LDC modules 14 Dl 0.65 
Revise LDC modules 14 2.50 0.76 
Use the LDC framework rubric 14 2.50 0.76 
Use students’ products to inform my instruction 14 2.50 0.76 


Provide helpful feedback to students about their 
writing 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C45 


Perceptions of Teacher Collaboration during LDC Implementation (Pennsylvania Science) 


Question n Mean Std Dev 


I would describe my LDC colleagues as collaborative 6 2.83 0.41 


Collaboration with my LDC colleagues helps me ... 


More effectively use the LDC framework 6 2.67 0.52 
Better support student learning 6 2.50 0.84 
Develop LDC modules 6 2.67 0.52 
Teach LDC modules 6 2.67 0.52 
Revise LDC modules 6 2.67 0.52 
Use the LDC framework rubric 6 2.67 0.52 
Use students’ products to inform my instruction 6 2.50 0.84 
Provide helpful feedback to students about their 6 2.00 1.27 


writing 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C46 


Perceptions of Teacher Collaboration during LDC Implementation (Pennsylvania Social Studies) 


Question n Mean Std Dev 
I would describe my LDC colleagues as collaborative 7 2.51} 0.79 
Collaboration with my LDC colleagues helps me ... 
More effectively use the LDC framework 7 2.57 0.79 
Better support student learning 7 2.57 0.79 
Develop LDC modules 7 2.57 0.79 
Teach LDC modules Th PE! 0.79 
Revise LDC modules si) 2.43 0.79 
Use the LDC framework rubric 7 2 0.79 
Use students’ products to inform my instruction y/ 2.14 0.90 
Provide helpful feedback to students about their 7 2414 0.90 


writing 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C47 
Teacher Perceptions of Efficacy in Teaching LDC Modules 


Question n Mean Std Dev 


Kentucky Science 


I knew what skills my students needed in order to complete 


the teaching task i 20 ee 
I knew the type of mini-tasks to give my students to prepare 

12 2.67 0.49 
them to complete the template task 
I understood how to use the LDC instructional ladder 12 2.42 0.51 


Based on the information collected from using the LDC 
modules, I adjusted my instruction to meet the needs of 12 2.58 0.67 
individual students 


Kentucky Social Studies 


I knew what skills my students needed in order to complete 


the teaching task “4 223 Oe 
I knew the type of mini-tasks to give my students to prepare 

14 2.21 0.80 
them to complete the template task 
I understood how to use the LDC instructional ladder 14 2.07 1.07 


Based on the information collected from using the LDC 
modules, I adjusted my instruction to meet the needs of 14 2.07 1.00 
individual students 


Pennsylvania Science 


I knew what skills my students needed in order to complete 


the teaching task : 250 OS? 
I knew the type of mini-tasks to give my students to prepare 6 2.50 0.55 
them to complete the template task ‘ 

I understood how to use the LDC instructional ladder 6 2.00 1.10 


Based on the information collected from using the LDC 
modules, I adjusted my instruction to meet the needs of 6 2.50 0.84 
individual students 


Pennsylvania Social Studies 


I knew what skills my students needed in order to complete 


the teaching task i aoe 038 
I knew the type of mini-tasks to give my students to prepare 7 271 0.76 
them to complete the template task : : 

I understood how to use the LDC instructional ladder 7 2.14 1.07 


Based on the information collected from using the LDC 
modules, I adjusted my instruction to meet the needs of 7 2.43 0.53 
individual students 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C48 
Potential Barriers to Use of LDC Modules (Kentucky Science) 


Barriers n Mean Std Dev 
I had sufficient time to prepare to teach modules 12 1.83 1.12 
I felt adequately prepared to effectively use modules 12 2.17 0.72 
It is difficult to find the time to respond to student writing 12 2.75 0.45 
I am unsure about how best to give productive feedback to student 2 1.75 0.97 
writing 
Using the LDC modules takes too much time away from covering 

; : . 12 1.67 0.99 

required curriculum topics 
It is challenging for me to find content-rich reading materials at my 12 2.50 0.80 
students’ reading level 
It is challenging for me to find the time to develop modules 12 2.50 0.52 
Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
Table C49 
Potential Barriers to Use of LDC Modules (Kentucky Social Studies) 

Barriers n Mean Std Dev 
I had sufficient time to prepare to teach modules 14 2.21 0.80 
I felt adequately prepared to effectively use modules 14 2.00 0.96 
It is difficult to find the time to respond to student writing 14 2.36 0.84 
I am unsure about how best to give productive feedback to student 14 1.79 112 
writing ‘ ‘ 
Using the LDC modules takes too much time away from covering 14 17] 0.91 
required curriculum topics ; ; 
It is challenging for me to find content-rich reading materials at my 

; ; 14 1.21 0.98 

students’ reading level 
It is challenging for me to find the time to develop modules 14 2.07 1.00 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C50 


Potential Barriers to Use of LDC Modules (Pennsylvania Science) 


Barriers n Mean Std Dev 
I had sufficient time to prepare to teach modules 6 1.33 1.21 
I felt adequately prepared to effectively use modules 6 2.00 0.63 
It is difficult to find the time to respond to student writing 6 2.50 0.55 
I am unsure about how best to give productive feedback to student 6 1.33 0.82 
writing ; ; 
Using the LDC modules takes too much time away from covering 
; : . 6 1.67 0.52 
required curriculum topics 
It is challenging for me to find content-rich reading materials at my 
; ; 6 2.00 0.89 
students’ reading level 
It is challenging for me to find the time to develop modules 6 2.33 0.82 
Note. Scale is 0 = disagree, | = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
Table C51 
Potential Barriers to Use of LDC Modules (Pennsylvania Social Studies) 
Barriers n Mean Std Dev 
I had sufficient time to prepare to teach modules 7 1.86 1.35 
I felt adequately prepared to effectively use modules 7 2.14 0.90 
It is difficult to find the time to respond to student writing 7 2.00 1.12 
I am unsure about how best to give productive feedback to student 7 071 0.95 
writing 
Using the LDC modules takes too much time away from covering 7 1.29 Li 
required curriculum topics : ‘ 
It is challenging for me to find content-rich reading materials at my 
> ; 7 1.29 1.38 
students’ reading level 
It is challenging for me to find the time to develop modules 7 1.71 0.95 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


a0 


Table C52 
Teacher Perceptions of Whether LDC Helped Them Meet Instructional Goals (Kentucky Science; n = 12) 


Question N Yes (%) n No (%) 
Find effective strategies for teaching my subject content 9 75.0 3 25.0 
Learn new ways to include formative assessment in my classes 7 58.3 5 41.7 
Develop new ways to teach literacy skills in my content area 10 83.3 2 16.7 
Learn detailed information about students’ literacy strengths and 

11 91.7 1 8.3 

weaknesses 
Provide students with more detailed feedback about their 9 75.0 3 250 
writing 
Implement the CCSS 10 83.3 2 16.7 
Increase the rigor of writing assessments 11 91.7 1 8.3 
Better engage students 5 41.7 7 58.3 
Table C53 


Teacher Perceptions of Whether LDC Helped Them Meet Instructional Goals (Kentucky Social Studies; n = 14) 


Question n Yes (%) n No (%) 
Find effective strategies for teaching my subject content 10 71.4 4 28.6 
Learn new ways to include formative assessment in my classes 9 64.3 5 35.7 
Develop new ways to teach literacy skills in my content area 11 78.6 3 21.4 
Learn detailed information about students’ literacy strengths and 12 85.7 2 143 
weaknesses 
Provide students with more detailed feedback about their 2 85.7 2 143 
writing 
Implement the CCSS 9 64.3 5 35.7 
Increase the rigor of writing assessments 14 100.0 0 0.0 
Better engage students 7 50.0 7 50.0 
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Table C54 


Teacher Perceptions of Whether LDC Helped Them Meet Instructional Goals (Pennsylvania Science; n = 6) 


Question Yes (%) N No (%) 
Find effective strategies for teaching my subject content 50.0 3 50.0 
Learn new ways to include formative assessment in my classes 83.3 1 16.7 
Develop new ways to teach literacy skills in my content area 100.0 0 0.0 
Learn detailed information about students’ literacy strengths and 

83.3 1 16.7 

weaknesses 
Provide students with more detailed feedback about their 66.7 2 333 
writing 
Implement the CCSS 100.0 0 0.0 
Increase the rigor of writing assessments 100.0 0 0.0 
Better engage students 33.3 4 66.7 


Table C55 
Teacher Perceptions of Whether LDC Helped Them Meet Instructional Goals (Pennsylvania Social Studies; 
n=7) 

Question Yes (%) n No (%) 
Find effective strategies for teaching my subject content 57.1 3 42.9 
Learn new ways to include formative assessment in my classes 71.4 2 28.6 
Develop new ways to teach literacy skills in my content area 85.7 1 14.3 
Learn detailed information about students’ literacy strengths and 

71.4 2 28.6 

weaknesses 
Provide students with more detailed feedback about their 57] 3 42.9 
writing 
Implement the CCSS 100.0 0 0,0 
Increase the rigor of writing assessments 85.7 1 14.3 
Better engage students 71.4 2 28.6 
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Table C56 


Teacher Perceptions of the Effectiveness of LDC (Kentucky Science) 


Question n Mean Std Dev 

Improving students' literacy skills 12 2.25 0.62 
Providing a curricular resource for teachers to address the CCSS 12 2.58 0.67 
Encouraging science and social studies teachers to teach literacy 2 2.42 0.67 
skills 

Encouraging secondary school teachers to teach literacy skills 12 2.42 0.67 
Making instruction more engaging for the students. 12 1.58 0.79 
Using formative assessment to identify student strengths and 2 1.92 0.79 


weaknesses to inform instruction 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C57 
Teacher Perceptions of the Effectiveness of LDC (Kentucky Social Studies) 


Question n Mean Std Dev 

Improving students' literacy skills 14 2.07 0.83 
Providing a curricular resource for teachers to address the CCSS 14 2.07 0.92 
Encouraging science and social studies teachers to teach literacy 14 221 0.80 
skills 

Encouraging secondary school teachers to teach literacy skills 14 2.29 0.83 
Making instruction more engaging for the students. 14 2.00 0.96 
Using formative assessment to identify student strengths and 14 2.00 0.96 


weaknesses to inform instruction 


Note. Scale is 0 = disagree, | = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C58 


Teacher Perceptions of the Effectiveness of LDC (Pennsylvania Science) 


Question n Mean Std Dev 
Improving students' literacy skills 6 2.67 0.52 
Providing a curricular resource for teachers to address the CCSS 6 2.33 0.82 
Encouraging science and social studies teachers to teach literacy 6 2.50 0.55 
skills 
Encouraging secondary school teachers to teach literacy skills 6 2.50 0.55 
Making instruction more engaging for the students. 6 Cale, 0.41 


Using formative assessment to identify student strengths and 


weaknesses to inform instruction : ae O32 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C59 


Teacher Perceptions of the Effectiveness of LDC (Pennsylvania Social Studies) 


Question n Mean Std Dev 
Improving students' literacy skills 7 2.43 0.79 
Providing a curricular resource for teachers to address the CCSS 7 2.86 0.38 
Encouraging science and social studies teachers to teach literacy 7 243 0.79 
skills 
Encouraging secondary school teachers to teach literacy skills 7 2.57 0.79 
Making instruction more engaging for the students. 7 2.14 0.69 


Using formative assessment to identify student strengths and 


weaknesses to inform instruction U a O82 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C60 
Student Engagement during LDC Module Instruction 


Effect on engagement n LDC teachers (“%) 


Kentucky Science (7 = 12) 


More engaged 0 0.0 

Same level of engagement 9 75.0 

Less engaged 3 25.0 
Kentucky Social Studies (n = 14) 

More engaged 4 28.6 

Same level of engagement 7 50.0 

Less engaged 3 21.4 
Pennsylvania Science (n = 6) 

More engaged 2 33.3 

Same level of engagement 4 66.7 

Less engaged 0 0.0 
Pennsylvania Social Studies (1 = 7) 

More engaged 2 28.6 

Same level of engagement 5 71.4 

Less engaged 0 0.0 
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Table C6l 


Perceptions of Student Success on LDC Tasks 


Strategies n No success Little Some A great deal 
(%) success (%) success(%) of success 
(%) 


Kentucky Science 


Reading mini-tasks built into the 12 0.0 0.0 66.7 33.3 

instructional ladder 

Writing mini-tasks built into the 12 0.0 0.0 58.3 41.7 

instructional ladder 

Final writing task 12 0.0 0.0 58.3 41.7 
Kentucky Social Studies 

Reading mini-tasks built into the 14 0.0 50.0 42.9 7.1 

instructional ladder 

Writing mini-tasks built into the 14 0.0 28.6 64.3 7.1 

instructional ladder 

Final writing task 14 0.0 354 57.1 7.1 
Pennsylvania Science 

Reading mini-tasks built into the 6 0.0 0.0 66.7 33.3 

instructional ladder 

Writing mini-tasks built into the 6 0.0. 0.0 66.7 33.3 

instructional ladder 

Final writing task 6 0.0 0.0. 66.7 33.3 
Pennsylvania Social Studies 

Reading mini-tasks built into the 7 0.0 14.3 57.1 28.6 

instructional ladder 

Writing mini-tasks built into the 7 0.0 0.0 71.4 28.6 

instructional ladder 

Final writing task 7 0.0 0.0 57.1 42.9 
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Table C62 


Teacher Perceptions on Student Impact of LDC Modules 


Student impact n Mean Std Dev 


Kentucky Science 


Resulted in higher quality student writing 12 2.17 0.577 

Supporting my students’ college-readiness 12 2.33 0.888 
Kentucky Social Studies 

Resulted in higher quality student writing 14 2.14 0.949 

Supporting my students’ college-readiness 14 2.21 0.699 
Pennsylvania Science 

Resulted in higher quality student writing 6 2.17 0.753 

Supporting my students’ college-readiness 6 2.33 0.516 


Pennsylvania Social Studies 
Resulted in higher quality student writing 7 2.14 0.900 
Supporting my students’ college-readiness 7 AT 0.535 


Note. Scale is 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C63 


Teacher Perceptions on Student Impact during Most Recent LDC Module 


Student impact n Yes (%) n No (%) 


Kentucky Science ( = 12) 


Majority of students improved their understanding of content 10 83.3 2 16.7 

Majority of students improved their literacy skills 10 83.3 2 16.7 
Kentucky Social Studies (1 = 14) 

Majority of students improved their understanding of content 12 85.7 2 14.3 

Majority of students improved their literacy skills 11 78.6 3 21.4 
Pennsylvania Science (n = 6) 

Majority of students improved their understanding of content 6 100.0 0 0.0 

Majority of students improved their literacy skills 5 83.3 1 16.7 
Pennsylvania Social Studies (n = 7) 

Majority of students improved their understanding of content 7 100.0 0 0.0 

Majority of students improved their literacy skills 4 541 3 42.9 
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Appendix D: 
Descriptive Analyses of Kentucky and Pennsylvania LDC Logs 


Table D1 


Average Percent of Time Spent on Different Classroom Activities (Kentucky — Science) 


Module one Module two Overall 

Components n Mean Std Dev n Mean Std Dev N Mean _— Std Dev 
Lecture on subject 7 7.47 5.98 6 8.25 9.96 13 7.83 7.70 
matter content 
Mini-Lessons 7 ~~ 6.23 2.62 6 3.57 2.34 13 5.00 2.76 
Explicit strategy 7 12.61 3.90 6 = 8.88 4.48 13 10.89 4.44 
instruction 
Whole-class discussion 7 7.94 5.04 6 13.02 9.39 13. 10.29 ESA 
Small group work 7 9.62 5.67 6 1.67 4.08 13 5.95 6.33 
Pair/share 7 3.78 525 6 1.63 2.58 13 2.78 4.22 
Independent 7 40.31 9.97 6 50.58 10.18 13. 45.05 11.01 
reading/writing 
Student presentations 7 3.91 6.63 6 3.60 5.35 13 3.77 5.82 
Other 7 8.12 8.53 6 68.81 12.75 13 8.44 10.21 


Note. Means represent percents. 


Table D2 


Average Percent of Time Spent on Different Classroom Activities (Kentucky — Social Studies) 


Module one Module two Overall 

Components n Mean Std Dev n Mean Std Dev N Mean _— Std Dev 
Lecture on subject 11 6.12 4.14 11 9.58 5.81 22 7.85 5.24 
matter content 
Mini-Lessons 11 6.65 4.67 11 5.25 5.75 22 5.95 5.16 
Explicit strategy 11 10.40 6.32 11 4.48 4.88 22 7.44 6.29 
instruction 
Whole-class discussion 11 11.37 6.80 11 10.51 7.96 22 10.94 7.24 
Small group work 11 11.48 14.17 11 5.16 6.29 22 8.32 11.18 
Pair/share 11 6.44 6.85 11 5.19 6.24 22 5.81 6.42 
Independent 11 48.14 15.28 11 52.40 26.69 22 50.27 21.33 
reading/writing 
Student presentations 11 1.90 2.99 11 1.67 3.73 22 1.78 3.30 
Other 11 5.31 8.30 11 5.76 10.55 22 5:53 9.27 
Note. Means represent percents. 
Table D3 
Average Percent of Time Spent on Different Classroom Activities (Pennsylvania - Science) 

Module one Module two Overall 

Components n Mean Std Dev n Mean Std Dev N Mean _— Std Dev 
Lecture on subject 10 4.84 6.07 8 3.42 3.91 18 4.21 5.13 
matter content 
Mini-Lessons 10 7.16 4.85 8 4.97 3.35 18 6.19 4.28 
Explicit strategy 10 7.23 5.41 8 5.21 4.57 18 6.33 5.02 
instruction 
Whole-class discussion 10 Tet 8.48 8 6.55 4.30 18 7.23 6.79 
Small group work 10 =2.06 3.11 8 5.51 6.77 18 3.59 5.20 
Pair/share 10 4.60 6.12 8 7.30 9.87 18 5.80 7.87 
Independent 10 60.70 19.82 8 60.36 15.92 18 60.55 17.67 
reading/writing 
Student presentations 10 2.13 6.72 8 0.00 0.00 18 1.18 5.01 
Other 10 3.51 6.40 8 6.68 9.63 18 4.92 7.91 


Note. Means represent percents. 


Table D4 


Average Percent of Time Spent on Different Classroom Activities (Pennsylvania — Social Studies) 


Module one Module two Overall 

Components n Mean Std Dev n Mean Std Dev N Mean _— Std Dev 
Lecture on subject 10 5.83 3.71 9 6.44 5.16 19 6.12 4.34 
matter content 
Mini-Lessons 10 8.26 6.50 9 8.90 6.20 19 8.56 6.19 
Explicit strategy 10 7.08 5.10 9 7.11 7.07 19 7.10 5.93 
instruction 
Whole-class discussion 10 7.05 3.86 9 8.92 8.57 19 7.94 6.40 
Small group work 10 10.31 13.65 9 7.30 8.65 19 8.89 11.35 
Pair/share 10 8.90 7.90 9 557 4.75 19 7.32 6.65 
Independent 10 44.55 13.10 9 50.19 21.70 19 47.22 17.42 
reading/writing 
Student presentations 10 2.07 3.99 9 1.59 4.76 19 1.84 4.25 
Other 10 5.96 8.74 9 3.97 6.26 19 5.02 7.52 


Note. Means represent percents. 


Table D5 


Reading Skills Students Worked on the Day of the Log (Kentucky — Science) 


Module one Module two Overall 

Skills n Mean Std Dev n Mean Std Dev N Mean — Std Dev 
Independent reading/ 6 0.94 0.80 5 1.80 0.30 11 1.33 0.75 
research 
Making predictions/ 6 860.44 0.39 4 0.83 0.88 10  ~=0.60 0.62 
previewing 
Summarizing 6 1.72 0.44 5 1.60 0.55 11 1.67 0.47 
important points 
Note-taking/ 6 1.83 0.41 5 1.80 0.45 11 1.82 0.40 
annotation 
Identifying/ defining 6 1.25 0.76 5 1.13 0.51 11 1.20 0.63 
vocabulary 
Analyzing text 5 0.53 0.77 5 0.80 0.84 10 ~=—-0.67 0.77 
structure (e.g., how 
part relates to whole) 
Interpreting 6 0.17 0.41 5 0.60 0.60 11 0.36 0.53 
information from 
graphical text 
Distinguishing fact 6  ~—0.00 0.00 5 0.20 0.45 11 0.09 0.30 
from opinion 
Drawing conclusions 6 = 0.67 0.52 5 0.93 0.80 11 0.79 0.64 
from textual evidence 
Citing textual evidence 6 = 0.56 0.78 5 0.93 0.92 11 0.73 0.83 
to support claims 
Evaluating strength/ 6 ~=0.06 0.14 5 0.73 0.83 11 0.36 0.64 
weakness of evidence 
Comparing arguments 6 0.17 0.41 5 0.87 1.04 11 0.48 0.81 
in two or more texts 
Examining author's 6 0.17 0.41 5 0.40 0.89 11 0.27 0.65 
perspective/ bias 
Examining rhetorical 6 0.00 0.00 5 0.00 0.00 11 0.00 0.00 
devices 
Other 3 0.00 0.00 5 0.00 0.00 8 0.00 0.00 


Note. Scale is 0=not today, 1=touched on briefly, 2=focus of instruction. 


Table D6 
Reading Skills Students Worked on the Day of the Log (Kentucky — Social Studies) 


Module one Module two Overall 

Skills n Mean Std Dev n Mean = Std Dev N Mean — Std Dev 
Independent reading/ 10 1.63 0.48 10 1.80 0.42 20 1.72 0.45 
research 
Making predictions/ 9 0.97 0.89 7 ~~ 0.79 0.39 16 = 0.89 0.70 
previewing 
Summarizing 10 1.63 0.46 10 1.75 0.42 20 1.69 0.44 
important points 
Note-taking/ 11 1.35 0.90 9 1.72 0.44 20 1.52 0.73 
annotation 
Identifying/ defining 9 1.13 0.78 8 1.56 0.62 17 1.34 0.72 
vocabulary 
Analyzing text 9 0.80 0.77 8 1.00 0.76 170.89 0.75 
structure (e.g., how 
part relates to whole) 
Interpreting 11 1.04 0.97 8 1.31 0.80 19 1.15 0.89 
information from 
graphical text 
Distinguishing fact 9 0.94 0.81 7 ~~ 0.79 0.91 16 = 0.88 0.83 
from opinion 
Drawing conclusions 10 1.54 0.69 8 1.63 0.44 18 1.58 0.58 
from textual evidence 
Citing textual evidence 9 1.23 0.86 8 1.63 0.44 17 1.42 0.70 
to support claims 
Evaluating strength/ 10 1.20 0.79 7 0.71 0.70 17 1.00 0.77 
weakness of evidence 
Comparing arguments 9  ~=0.89 0.93 8 1.00 0.93 17 0.94 0.90 
in two or more texts 
Examining author's 10 1.12 0.99 8 0.94 0.94 18 1.04 0.95 
perspective/ bias 
Examining rhetorical 8 0.61 0.70 6  ~=0.50 0.84 14 0.56 0.73 
devices 
Other 1 0.00 : 1 1.00 ; 2 0.50 0.71 


Note. Scale is 0=not today, 1=touched on briefly, 2=focus of instruction. 


Table D7 


Reading Skills Students Worked on the Day of the Log (Pennsylvania — Science) 


Module one Module two Overall 

Skills n Mean = Std Dev n Mean = Std Dev N Mean — Std Dev 
Independent reading/ 10 1.80 0.32 7 ~~ 2.00 0.00 17 1.88 0.26 
research 
Making predictions/ 10 =0.52 0.52 7 0.55 0.74 17. — 0.53 0.60 
previewing 
Summarizing 10 1.45 0.65 7 1.14 0.63 17 1.32 0.64 
important points 
Note-taking/ 9 1.57 0.58 7 1.81 0.24 16 1.68 0.47 
annotation 
Identifying/ defining 10 1.35 0.70 i] 1.05 0.88 17 1.23 0.77 
vocabulary 
Analyzing text 9 1.00 0.77 7 ~~ 0.50 0.76 16 ~—- 0.78 0.78 
structure (e.g., how 
part relates to whole) 
Interpreting 9 0.59 0.80 6  ~=0.50 0.84 15 0.56 0.78 
information from 
graphical text 
Distinguishing fact 9 0.59 0.76 7 ~~ 0.93 0.93 16 390.74 0.83 
from opinion 
Drawing conclusions 9 1.02 0.77 i 1.36 0.48 16 1.17 0.66 
from textual evidence 
Citing textual evidence 9 0.70 0.77 7 1.19 0.84 16 0.92 0.81 
to support claims 
Evaluating strength/ 9 0.59 0.66 7 1.00 1.00 16 =0.77 0.82 
weakness of evidence 
Comparing arguments 9 =0.48 0.71 6 0.33 0.82 15 0.42 0.73 
in two or more texts 
Examining author's 9 0.39 0.70 6  ~=0.50 0.84 15 0.43 0.73 
perspective/ bias 
Examining rhetorical 9 0.06 0.17 6 0.00 0.00 15 0.03 0.13 
devices 
Other 6 ~—-0..00 0.00 6 0.00 0.00 12. ~—0.00 0.00 


Note. Scale is 0=not today, 1=touched on briefly, 2=focus of instruction. 


Table D8 
Reading Skills Students Worked on the Day of the Log (Pennsylvania — Social Studies) 


Module one Module two Overall 

Skills n Mean Std Dev n Mean = Std Dev N Mean — Std Dev 
Independent reading/ 10 1.87 0.32 9 1.75 0.43 19 1.81 0.37 
research 
Making predictions/ 9 0.50 0.67 9 0.57 0.71 18 0.54 0.67 
previewing 
Summarizing 9 1.70 0.42 9 1.21 0.60 18 1.46 0.56 
important points 
Note-taking/ 10 1.77 0.48 9 1.43 0.68 19 1.61 0.59 
annotation 
Identifying/ defining 9 0.85 0.48 9 0.77 0.66 18 0.81 0.56 
vocabulary 
Analyzing text 9 0.60 0.81 9 0.72 0.67 18 0.66 0.72 
structure (e.g., how 
part relates to whole) 
Interpreting 9 0.43 0.49 9 0.56 0.53 18 ~=0.49 0.50 
information from 
graphical text 
Distinguishing fact 9 ~=0.48 0.40 9 =0.78 0.67 18 0.63 0.56 
from opinion 
Drawing conclusions 9 1.16 0.73 9 1.06 0.53 18 1.11 0.62 
from textual evidence 
Citing textual evidence 10 = 0.95 0.80 9 1.08 0.73 19 1.01 0.75 
to support claims 
Evaluating strength/ 9 0.99 0.42 9 1.02 0.68 18 1.00 0.55 
weakness of evidence 
Comparing arguments 9 0.33 0.43 9 0.38 0.47 18 0.36 0.44 
in two or more texts 
Examining author's 9 0.50 0.44 9 0.50 0.50 18 0.50 0.46 
perspective/ bias 
Examining rhetorical 9 0.19 0.38 9 0.33 0.50 18 0.26 0.44 
devices 
Other 5 0.00 0.00 5 0.00 0.00 10. ~=0.00 0.00 


Note. Scale is 0=not today, 1=touched on briefly, 2=focus of instruction. 


Table D9 


Extent Teachers Relied on Different Strategies to Assess Student Understanding during the Reading Component 


(Kentucky — Science) 


Module one Module two Overall 

Strategies n Mean Std Dev n Mean Std Dev n Mean Std Dev 
Listened as students 4 1.13 0.85 3 0.00 0.00 v4 0.64 0.85 
discussed text with 

peers 

Circulated and 4 1.25 0.50 3 1.17 0.76 7 1.21 0.57 
reviewed student notes 
Reviewed peers' 3 0.00 0.00 2 0.00 0.00 5 0.00 0.00 
feedback 
Collected and 3 0.00 0.00 2 1.00 1.41 5 0.40 0.89 
reviewed student 
written responses 
and/or graphic 
organizers 
Asked students to 3 1.33 0.76 2 0.50 0.71 5 1.00 0.79 
answer oral questions 
Listened to students 3 0.83 1.04 2 1.00 1.41 5 0.90 1.02 
thinking aloud while 
reading 
Led whole-class 4 1.00 0.41 2, 0.50 0.71 6 0.83 0.52 
discussion 
Listened to student 4 1.00 0.00 2 1.00 0.00 6 1.00 0.00 
questions 
Assigned a quiz 3 0.00 0.00 2 0.00 0.00 5 0.00 0.00 
Graded student work 3 0.00 0.00 2 0.50 0.71 5 0.20 0.45 
Exit slips 3 0.00 0.00 2: 0.00 0.00 5 0.00 0.00 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 


Table D10 


Extent Teachers Relied on Different Strategies to Assess Student Understanding during the Reading Component 
(Kentucky — Social Studies) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean = Std Dev n Mean Std Dev 
Listened as students 9 1.25 0.39 6 1.42 0.66 15 1.32 0.50 
discussed text with 

peers 

Circulated and 10 1.58 0.44 7 1.50 0.65 17 1.55 0.52 
reviewed student notes 
Reviewed peers' 7 ~~ 0.23 0.41 6  ~=0.50 0.84 13 0.35 0.63 
feedback 
Collected and 8 1.12 0.79 7 1.07 0.73 15 1.10 0.74 
reviewed student 
written responses 
and/or graphic 
organizers 
Asked students to 11 1.51 0.50 6 1.50 0.55 17 1.51 0.50 
answer oral questions 
Listened to students 8 1.05 0.69 6 1.08 0.80 14 1.06 0.71 
thinking aloud while 
reading 
Led whole-class 10 1.32 0.64 6 1.58 0.80 16 1.42 0.69 
discussion 
Listened to student 9 1.41 0.47 6 1.67 0.41 15 1.51 0.45 
questions 
Assigned a quiz 7 ~~ 0.00 0.00 6 ~=—-0..00 0.00 13 0.00 0.00 
Graded student work 7 ~~ ~0.40 0.50 7 ~~ 0.36 0.75 14. 0.38 0.61 
Exit slips 8 0.28 0.70 6  ~=0.67 0.82 14. 0.44 0.75 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 


Table D11 


Extent Teachers Relied on Different Strategies to Assess Student Understanding during the Reading Component 
(Pennsylvania — Science) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean Std Dev n Mean Std Dev 
Listened as students 8 0.71 0.57 6 1.03 0.69 14. 0.85 0.62 
discussed text with 

peers 

Circulated and 8 1.65 0.44 6 1.61 0.49 14 1.63 0.44 
reviewed student notes 
Reviewed peers' 8 0.35 0.52 5 0.00 0.00 13 0.22 0.43 
feedback 
Collected and 8 0.25 0.46 6 ~=0.50 0.84 14. 0.36 0.63 
reviewed student 
written responses 
and/or graphic 
organizers 
Asked students to 8 0.96 0.53 6 30.97 0.73 14. 0.96 0.60 
answer oral questions 
Listened to students 8 0.81 0.70 6 1.00 0.89 14. 0.89 0.76 
thinking aloud while 
reading 
Led whole-class 8 0.42 0.38 6 1.22 0.75 14. 0.76 0.68 
discussion 
Listened to student 8 0.90 0.77 6 1.39 0.71 14 1.11 0.76 
questions 
Assigned a quiz 8 0.00 0.00 5 0.00 0.00 13 0.00 0.00 
Graded student work 8 0.00 0.00 6  ~=0.50 0.84 14. 0.21 0.58 
Exit slips 8 0.21 0.40 5 0.00 0.00 13 0.13 0.32 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D12 


Extent Teachers Relied on Different Strategies to Assess Student Understanding during the Reading Component 
(Pennsylvania — Social Studies) 


Module one Module two Overall 

Strategies n Mean Std Dev n Mean Std Dev n Mean Std Dev 
Listened as students 9 1.39 0.75 8 1.33 0.89 17 1.36 0.79 
discussed text with 

peers 

Circulated and 10 1.57 0.50 8 1.72 0.70 18 1.63 0.58 
reviewed student notes 
Reviewed peers' 9 0.56 0.47 8 0.70 0.88 17 0.62 0.67 
feedback 
Collected and 8 0.88 0.78 8 1.07 0.85 16 0.97 0.79 
reviewed student 
written responses 
and/or graphic 
organizers 
Asked students to 10 1.62 0.46 8 1.51 0.72 18 1.57 0.57 
answer oral questions 
Listened to students 9 1.06 0.92 8 1.04 0.95 17 1.05 0.90 
thinking aloud while 
reading 
Led whole-class 9 1.17 0.71 8 0.91 0.65 17 1.04 0.67 
discussion 
Listened to student 9 1.50 0.44 8 1.38 0.44 17 1.44 0.43 
questions 
Assigned a quiz 9 0.00 0.00 8 0.03 0.09 17 0.01 0.06 
Graded student work 9 0.22 0.37 8 0.53 0.74 17 0.37 0.58 
Exit slips 9 0.37 0.73 8 0.25 0.71 17 0.31 0.70 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D13 


Strategies Used when Discovered Student Misunderstandings about Reading (Kentucky — Science) 


Module one Module two Overall 

Strategies n Mean Std Dev n Mean Std Dev n Mean Std Dev 
One-on-one 2 0.50 0.71 3 1.00 1.00 0.80 0.84 
conference to provide 
feedback 
Asked peer to provide 2 0.00 0.00 2 0.00 0.00 0.00 0.00 
feedback 
Stopped class and 3 1.17 0.29 3 0.67 0.58 0.92 0.49 
modeled strategy 
Wrote specific 2 0.00 0.00 2 1.00 1.41 0.50 1.00 
comments on student 

work 

Scheduled in-class 2 0.00 0.00 2: 1.00 1.41 0.50 1.00 
workshop time 
Devoted time in lesson 3 0.50 0.50 2 0.50 0.71 0.50 0.50 
for students to use 
feedback 
Grouped students 2 0.00 0.00 2 1.00 0.00 0.50 0.58 
together on a "need" 
basis for targeted 
instruction 
Offered student a hint 3 0.50 0.50 3 1.00 0.00 0.75 0.42 
or suggestion 
Gave student the 3 0.67 1.15 2 0.50 0.71 0.60 0.89 
answer 
Gave student more 2 1.00 0.00 2 1.00 0.00 1.00 0.00 
time to try again and 
self-correct 
Graded student work 2 0.00 0.00 2: 0.50 0.71 0.25 0.50 
Re-taught lesson 2 0.00 0.00 2 0.50 0.71 0.25 0.50 
segment 
Planned to review skill 2: 0.50 0.71 2 0.50 0.71 0.50 0.58 
in future lessons 
Other 2, 0.00 0.00 2 0.00 0.00 0.00 0.00 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent 
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Table D14 


Strategies Used when Discovered Student Misunderstandings about Reading (Kentucky — Social Studies) 


Module one Module two Overall 

Strategies n Mean Std Dev n Mean = Std Dev n Mean Std Dev 
One-on-one 10 1.19 0.75 7 1.21 0.57 17 1.20 0.66 
conference to provide 
feedback 
Asked peer to provide 9 0.73 0.83 7 0.79 0.57 16 0.76 0.71 
feedback 
Stopped class and 10 0.85 0.58 7 0.79 0.57 17 0.82 0.56 
modeled strategy 
Wrote specific 8 0.29 0.42 6 0.83 0.75 14 0.52 0.63 
comments on student 

work 

Scheduled in-class 8 0.38 0.74 6 0.33 0.82 14 0.36 0.74 
workshop time 
Devoted time in lesson 8 0.60 0.68 6 0.67 0.82 14 0.63 0.71 
for students to use 
feedback 
Grouped students 9 0.53 0.71 6 0.25 0.61 15 0.42 0.67 
together on a "need" 
basis for targeted 
instruction 
Offered student a hint 11 1.27 0.52 6 1.25 0.42 17 1.26 0.47 
or suggestion 
Gave student the 8 0.40 0.77 6 0.50 0.84 14 0.44 0.77 
answer 
Gave student more 10 0.90 0.57 6 0.92 0.80 16 0.91 0.64 
time to try again and 
self-correct 
Graded student work 8 0.38 0.52 6 0.17 0.41 14 0.29 0.47 
Re-taught lesson 8 0.44 0.50 6 0.33 0.52 14 0.39 0.49 
segment 
Planned to review skill 8 0.70 0.70 7 1.07 0.61 15 0.87 0.66 
in future lessons 
Other 2, 0.50 0.71 1 0.00 3 0.33 0.58 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent 
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Table D15 


Strategies Used when Discovered Student Misunderstandings about Reading (Pennsylvania — Science) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean = Std Dev n Mean Std Dev 
One-on-one 8 1.21 0.81 6 1.33 0.52 14 1.26 0.68 
conference to provide 
feedback 
Asked peer to provide 8 0.42 0.38 5 0.57 0.43 13 0.47 0.39 
feedback 
Stopped class and 8 0.88 0.61 6 0.89 0.69 14 0.88 0.62 
modeled strategy 
Wrote specific 8 0.10 0.20 4 0.00 0.00 12. 0.07 0.17 
comments on student 

work 

Scheduled in-class 8 0.42 0.64 6 0.75 0.88 14 0.56 0.74 
workshop time 
Devoted time in lesson 8 0.35 0.59 6 0.67 1.03 14 0.49 0.79 
for students to use 
feedback 
Grouped students 8 0.21 0.40 5 0.20 0.45 13 0.21 0.40 
together on a "need" 
basis for targeted 
instruction 
Offered student a hint 8 0.63 0.44 6 0.94 0.65 14 0.76 0.54 
or suggestion 
Gave student the 8 0.13 0.23 5 0.40 0.55 13 0.23 0.39 
answer 
Gave student more 8 0.81 0.26 6 0.83 0.98 14 0.82 0.64 
time to try again and 
self-correct 
Graded student work 8 0.00 0.00 6 0.42 0.80 14 0.18 0.54 
Re-taught lesson 8 0.15 0.27 6 0.42 0.80 14 0.26 0.55 
segment 
Planned to review skill 8 0.60 0.45 5 0.40 0.55 13 0.53 0.48 
in future lessons 
Other 6 0.00 0.00 4 0.00 0.00 10 0.00 0.00 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent 
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Table D16 


Strategies Used when Discovered Student Misunderstandings about Reading (Pennsylvania — Social Studies) 


Module one Module two Overall 

Strategies n Mean Std Dev n Mean = Std Dev n Mean Std Dev 
One-on-one 10 1.23 0.79 9 1.54 0.59 19 1.38 0.70 
conference to provide 
feedback 
Asked peer to provide 9 0.69 0.46 9 1.06 0.73 18 0.88 0.62 
feedback 
Stopped class and 9 0.99 0.52 9 0.89 0.65 18 0.94 0.57 
modeled strategy 
Wrote specific 9 0.69 0.61 9 0.26 0.43 18 0.47 0.56 
comments on student 

work 

Scheduled in-class 9 0.80 0.96 9 0.59 0.86 18 0.69 0.89 
workshop time 
Devoted time in lesson 9 0.72 0.87 9 1.01 0.66 18 0.87 0.77 
for students to use 
feedback 
Grouped students 9 0.78 0.90 9 0.57 0.77 18 0.68 0.82 
together on a "need" 
basis for targeted 
instruction 
Offered student a hint 10 1.34 0.63 9 1.54 0.49 19 1.43 0.56 
or suggestion 
Gave student the 9 0.30 0.42 9 0.14 0.33 18 0.22 0.38 
answer 
Gave student more 9 1.07 0.52 9 0.93 0.55 18 1.00 0.52 
time to try again and 
self-correct 
Graded student work 9 0.19 0.34 9 0.31 0.43 18 0.25 0.38 
Re-taught lesson 9 0.28 0.44 9 0.35 0.49 18 0.31 0.45 
segment 
Planned to review skill 9 0.78 0.73 9 0.50 0.71 18 0.64 0.71 
in future lessons 
Other 5 0.40 0.89 6 0.33 0.82 11 0.36 0.81 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent 
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Table D17 


Areas of Writing Students Worked on the Day of the Log (Kentucky — Science) 


Module one Module two Overall 

Areas n Mean Std Dev n Mean Std Dev n Mean Std Dev 
Generating ideas for 0.55 0.74 5 0.70 0.67 12 0.61 0.68 
writing 
Outlining 0.22 0.40 5 1.10 0.74 11 0.62 0.72 
Writing/text structure 1.44 0.74 6 1.17 0.98 13 1.31 0.84 
Formulating a thesis 0.85 0.72 5 1.60 0.89 11 1.19 0.86 
statement 
Formulating a counter- 0.00 0.00 5 1.00 1.00 11 0.45 0.82 
argument 
Writing an 1.45 0.61 5 1.60 0.65 12 1.51 0.60 
introduction 
Writing a conclusion 1.48 0.71 5 1.30 0.76 12 1.40 0.70 
Writing a body 1.44 0.55 5 1.40 0.65 12 1.42 0.57 
paragraph 
Using transitional 0.86 0.82 5 1.10 0.89 11 0.97 0.82 
words or phrases 
Incorporating 1.14 0.72 5 1.40 0.65 12 1625; 0.67 
quotes/evidence 
Style/ word choice/ 0.83 0.77 5 1.10 0.89 11 0.95 0.80 
syntax 
Grammar conventions 0.85 0.78 5 1.00 0.94 11 0.92 0.81 


Note. Scale is 0=not today, 1=touched on briefly, 2=focus of instruction. 
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Table D18 
Areas of Writing Students Worked on the Day of the Log (Kentucky — Social Studies) 


Module one Module two Overall 

Areas n Mean Std Dev n Mean Std Dev n Mean Std Dev 
Generating ideas for 9 1.03 0.54 8 0.95 0.80 17 0.99 0.65 
writing 
Outlining 8 1.00 0.64 8 60.91 0.86 16 0.95 0.74 
Writing/text structure 10 1.41 0.53 9 1.29 0.81 19 1.35 0.66 
Formulating a thesis 9 1.22 0.55 8 1.16 0.79 17 1.19 0.66 
statement 
Formulating a counter- 9 =0.78 0.72 8 0.78 0.90 170.78 0.79 
argument 
Writing an 11 1.57 0.48 10 1.58 0.75 21 1.57 0.60 
introduction 
Writing a conclusion 11 1.46 0.62 10 1.64 0.52 21 1.55. 0.56 
Writing a body 11 1.43 0.60 10 1.65 0.52 21 1.54 0.56 
paragraph 
Using transitional 8 0.97 0.69 8 1.61 0.46 16 1.29 0.66 
words or phrases 
Incorporating 11 1.39 0.63 9 1.52 0.66 20 1.45 0.63 
quotes/evidence 
Style/ word choice/ 9 1.08 0.47 9 1.43 0.66 18 1.25 0.58 
syntax 
Grammar conventions 9 0.98 0.41 9 1.14 0.69 18 1.06 0.56 


Note. Scale is 0=not today, 1=touched on briefly, 2=focus of instruction. 
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Table D19 
Areas of Writing Students Worked on the Day of the Log (Pennsylvania - Science) 


Module one Module two Overall 

Areas n Mean = Std Dev n Mean Std Dev n Mean Std Dev 
Generating ideas for 9 0.87 0.77 7 0.57 0.79 16 0.74 0.77 
writing 
Outlining 9 0.69 0.77 7 ~~ 0.40 0.73 16 =0.57 0.74 
Writing/text structure 9 1.16 0.78 7 1.71 0.39 16 1.40 0.68 
Formulating a thesis 8 0.60 0.73 7 0.86 0.90 15 0.72 0.80 
statement 
Formulating a counter- 9 =0.78 0.84 5 0.10 0.22 14 0.54 0.75 
argument 
Writing an 10 1.11 0.76 7 1.64 0.75 17 1.33 0.78 
introduction 
Writing a conclusion 10 1.17 0.70 6 1.42 0.80 16 1.26 0.72 
Writing a body 10 1.21 0.66 if 1.64 0.75 17 1.39 0.71 
paragraph 
Using transitional 9 1.10 0.71 7 1.36 0.85 16 1.21 0.76 
words or phrases 
Incorporating 9 1.32 0.70 7 1.39 0.70 16 1.35; 0.68 
quotes/evidence 
Style/ word choice/ 9 0.84 0.67 7 1.29 0.81 16 1.03 0.75 
syntax 
Grammar conventions 9 0.89 0.61 7 1.21 0.81 16 1.03 0.70 


Note. Scale is 0=not today, 1=touched on briefly, 2=focus of instruction. 
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Table D20 


Areas of Writing Students Worked on the Day of the Log (Pennsylvania — Social Studies) 


Module one Module two Overall 

Areas n Mean Std Dev n Mean Std Dev n Mean = Std Dev 
Generating ideas for 10 0.66 0.54 7 0.45 0.53 17 0.58 0.53 
writing 
Outlining 10 0.59 0.47 7 ~~ 0.67 0.43 170.62 0.44 
Writing/text structure 10 1.14 0.87 7 ~~ 0.95 0.82 17 1.06 0.83 
Formulating a thesis 10 0.84 0.66 ad 0.67 0.75 17 0.77 0.68 
statement 
Formulating a counter- 10 0.54 0.69 7 ~~ 0.00 0.00 17. 0.32 0.58 
argument 
Writing an 10 0.97 0.80 7 1.29 0.71 17 1.10 0.75 
introduction 
Writing a conclusion 10 ~=—0.86 0.91 7 1.00 0.51 17 — 0.92 0.76 
Writing a body 10 1.27 0.76 7 1.31 0.53 17 1.29 0.66 
paragraph 
Using transitional 10 1.03 0.82 7 1.14 0.61 17 1.08 0.72 
words or phrases 
Incorporating 10 1.31 0.62 7 1.07 0.73 17 1.21 0.66 
quotes/evidence 
Style/ word choice/ 10 1.07 0.64 7 ~~ 0.74 0.71 170.93 0.67 
syntax 
Grammar conventions 10 1.19 0.65 7 0.79 0.66 17 1.02 0.67 


Note. Scale is 0=not today, 1=touched on briefly, 2=focus of instruction. 


19 


Table D21 


Extent Teachers Relied on Different Strategies to Assess Student Understanding during Writing Component 


(Kentucky — Science) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean = Std Dev n Mean Std Dev 
Listened as students 5 0.80 0.84 5 0.40 0.55 10  ~=0.60 0.70 
discussed draft with 

peers 
Asked students to 5 1.00 0.94 5 0.40 0.89 10. ~=0.70 0.92 
provide feedback to 
each other 
Observed and 5 1.67 0.58 6 1.75 0.42 11 1.71 0.47 
reviewed student work 
Collected and 5 0.93 1.01 5 0.40 0.89 10. ~=—-0.67 0.94 
reviewed student 
writing exercises 
Asked students to 5 1.00 0.75 5 0.90 0.74 10 = 0.95 0.70 
answer oral questions 
Reviewed student 6 1.33 0.76 6 ~=0.67 0.82 12 1.00 0.83 
rough drafts 
Asked certain students 5 0.00 0.00 5 0.10 0.22 10 ~=—-0.05 0.16 
to present writing to 
class 

Assigned a quiz 5 0.00 0.00 5 0.00 0.00 10 ~=0.00 0.00 
Graded student work 5 0.40 0.55 5 0.00 0.00 10 ~=—-0.20 0.42 
Exit slips 5 0.00 0.00 5 0.00 0.00 10  ~=0.00 0.00 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D22 


Extent Teachers Relied on Different Strategies to Assess Student Understanding during Writing Component 


(Kentucky — Social Studies) 


Module one Module two Overall 

Strategies n Mean Std Dev n Mean = Std Dev n Mean Std Dev 
Listened as students 8 0.77 0.78 7 1.02 0.88 15 0.89 0.81 
discussed draft with 

peers 
Asked students to 10 1.18 0.67 8 1.15 0.77 18 1.16 0.69 
provide feedback to 
each other 
Observed and 10 1.58 0.40 9 1.72 0.44 19 1.65 0.42 
reviewed student work 
Collected and 9 1.06 0.85 7 1.19 0.84 16 1.11 0.82 
reviewed student 
writing exercises 
Asked students to 8 0.78 0.41 7 1.00 0.51 15 0.88 0.46 
answer oral questions 
Reviewed student 9 1.69 0.66 9 1.26 0.43 18 1.48 0.59 
rough drafts 
Asked certain students 8 0.47 0.76 7 ~~ 0.24 0.42 15 0.36 0.62 
to present writing to 
class 

Assigned a quiz 8 0.25 0.71 7 ~~ 0.00 0.00 15 0.13 0.52 
Graded student work 7 ~~ 0.29 0.39 8 0.35 0.69 15 0.32 0.55 
Exit slips 7 ~~ 0.00 0.00 7 ~~ 0.00 0.00 14 ~—-0..00 0.00 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D23 
Extent Teachers Relied on Different Strategies to Assess Student Understanding during Writing Component 


(Pennsylvania - Science) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean = Std Dev n Mean Std Dev 
Listened as students 7 1.13 0.74 6 1.08 0.66 13 1.11 0.68 
discussed draft with 

peers 
Asked students to 7 1.00 0.82 6 ~=0.50 0.55 13 0.77 0.73 
provide feedback to 
each other 
Observed and 7 1.65 0.46 6 1.42 0.80 13 1.54 0.62 
reviewed student work 
Collected and 7 ~~ 0.47 0.62 6 0.25 0.42 13 0.37 0.53 
reviewed student 
writing exercises 
Asked students to 7 ~~ 0.89 0.64 6 0.92 0.80 13 0.90 0.69 
answer oral questions 
Reviewed student 7 1.01 0.52 6 1.08 0.92 13 1.04 0.70 
rough drafts 
Asked certain students 7 ~~ 0.00 0.00 6 ~=0.33 0.82 13 0.15 0.55 
to present writing to 
class 

Assigned a quiz 7 ~~ 0.00 0.00 5 0.00 0.00 12 ~—0.00 0.00 
Graded student work 7 ~~ 0.00 0.00 5 0.30 0.45 12. 0.13 0.31 
Exit slips 7 ~~ 0.00 0.00 5 0.00 0.00 12. ~—0.00 0.00 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D24 
Extent Teachers Relied on Different Strategies to Assess Student Understanding during Writing Component 


(Pennsylvania — Social Studies) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean = Std Dev n Mean Std Dev 
Listened as students 10 1.13 0.74 7 0.86 0.75 17 1.02 0.73 
discussed draft with 

peers 
Asked students to 10 1.13 0.74 7 0.52 0.69 170.88 0.76 
provide feedback to 
each other 
Observed and 10 1.48 0.67 7 1.31 0.65 17 1.41 0.65 
reviewed student work 
Collected and 10 = 0.67 0.77 7 ~~ 0.57 0.45 170.63 0.64 
reviewed student 
writing exercises 
Asked students to 10 1.00 0.68 7 0.81 0.24 17. 0.92 0.54 
answer oral questions 
Reviewed student 10 1.22 0.75 7 1.05 0.77 17 1.15 0.74 
rough drafts 
Asked certain students 10 0.28 0.35 7 ~~ 0.29 0.49 170.28 0.40 
to present writing to 
class 

Assigned a quiz 10. ~—0.00 0.00 7 ~~ 0.00 0.00 17. ~—-:0.00 0.00 
Graded student work 10 0.33 0.72 7 ~~ 0.29 0.49 17. 0.31 0.62 
Exit slips 10 ~=0.20 0.63 7 ~~ 0.00 0.00 170.12 0.49 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D25 


Strategies Used when Discovered Student Misunderstandings about Writing (Kentucky - Science) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean = Std Dev n Mean Std Dev 
Organized peer-editing 4 0.50 0.58 4 0.25 0.50 0.38 0.52 
session 
Scheduled in-class 4 0.25 0.50 4 0.38 0.75 0.31 0.59 
workshop time 
Held one-on-one 4 1.21 0.63 5 1.00 0.61 1.09 0.59 
conference with 
student 
Devoted time in lesson 4 0.25 0.50 4 1.13 0.85 0.69 0.80 
for students to use 
feedback 
Grouped students 4 0.00 0.00 4 0.50 1.00 0.25 0.71 
together on "need" 
basis for targeted 
instruction 
Modeled skill using 4 0.42 0.50 4 1.00 0.82 0.71 0.70 
my own writing 
Demonstrated skill 4 0.25 0.50 4 0.38 0.48 0.31 0.46 
using student's writing 
Provided grammar 4 0.00 0.00 4 0.00 0.00 0.00 0.00 
mini-lessons 
Wrote specific 4 1.17 1.00 4 0.75 0.96 0.96 0.93 
comments on student 

work 

Had student revisit 5 1.07 0.92 4 0.88 0.25 0.98 0.68 
readings 
Offered student a hint 5 1.30 0.67 4 1.13 0.63 1.22 0.62 
or suggestion 
Gave student time to 4 1.46 0.42 4 1.38 0.75 1.42 0.56 
try again and self- 
correct 
Corrected student 4 0.58 0.50 4 0.63 0.48 0.60 0.45 
writing 
Graded student work 4 0.50 0.58 4 0.38 0.75 0.44 0.62 
Re-taught lesson 4 0.13 0.25 4 0.25 0.50 0.19 0.37 
segment 
Planned to review skill 4 0.33 0.47 4 0.00 0.00 0.17 0.36 
in future lessons 
Other 3 0.33 0.58 4 0.00 0.00 0.14 0.38 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D26 


Strategies Used when Discovered Student Misunderstandings about Writing (Kentucky - Social Studies) 


Module one Module two Overall 

Strategies n Mean Std Dev n Mean Std Dev n Mean Std Dev 
Organized peer-editing 9 0.86 0.65 7 ~~ ~=0.88 0.66 16 = 0.87 0.63 
session 
Scheduled in-class 11 1.27 0.66 7 0.86 0.75 18 1.11 0.71 
workshop time 
Held one-on-one 10 1.40 0.62 9 1.28 0.79 19 1.34 0.69 
conference with 
student 
Devoted time in lesson 8 0.58 0.62 7 1.14 0.86 15 0.84 0.77 
for students to use 
feedback 
Grouped students 9 0.65 0.73 8 0.90 0.68 17 0.76 0.70 
together on "need" 
basis for targeted 
instruction 
Modeled skill using 8 0.64 0.62 ah 0.76 0.65 15 0.69 0.62 
my own writing 
Demonstrated skill 10 0.82 0.88 7 0.69 0.78 17 0.76 0.82 
using student's writing 
Provided grammar 8 0.32 0.51 7 0.31 0.41 15 0.32 0.45 
mini-lessons 
Wrote specific 9 1.20 0.54 7 1.30 0.55 16 1.24 0.53 
comments on student 

work 

Had student revisit 10 1.08 0.55 7 1.29 0.50 17 1.16 0.53 
readings 
Offered student a hint 10 1.13 0.62 9 1.35 0.44 19 1.24 0.54 
or suggestion 
Gave student time to 10 1.21 0.60 9 1.57 0.39 19 1.38 0.53 
try again and self- 
correct 
Corrected student 10 1.18 0.71 8 1.21 0.85 18 1.19 0.75 
writing 
Graded student work 8 0.36 0.37 7 0.21 0.28 15 0.29 0.33 
Re-taught lesson 8 0.34 0.31 8 0.35 0.44 16 0.35 0.37 
segment 
Planned to review skill 8 0.30 0.38 8 0.75 0.58 16 0.53 0.53 
in future lessons 
Other 5 0.00 0.00 2 0.00 0.00 a. 0.00 0.00 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D27 


Strategies Used when Discovered Student Misunderstandings about Writing (Pennsylvania - Science) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean = Std Dev n Mean Std Dev 
Organized peer-editing 8 0.48 0.51 7 0.64 0.94 15 0.55 0.72 
session 
Scheduled in-class 7 0.86 0.82 7 0.57 0.98 14 0.72 0.88 
workshop time 
Held one-on-one 8 1.23 0.60 7 1.50 0.50 15 1.36 0.55 
conference with 
student 
Devoted time in lesson 7 0.87 0.50 7 0.86 0.75 14 0.86 0.61 
for students to use 
feedback 
Grouped students 7 0.19 0.33 a 0.50 0.76 14 0.34 0.59 
together on "need" 
basis for targeted 
instruction 
Modeled skill using 8 0.71 0.57 7 0.71 0.70 15 0.71 0.61 
my own writing 
Demonstrated skill 7 0.65 0.61 7 0.43 0.61 14 0.54 0.59 
using student's writing 
Provided grammar 7 ~~ 0.30 0.40 6 = 0.58 0.92 13 0.43 0.67 
mini-lessons 
Wrote specific 7 0.41 0.46 7 0.43 0.61 14 0.42 0.52 
comments on student 

work 

Had student revisit 8 0.84 0.35 i 1.00 0.82 15 0.92 0.60 
readings 
Offered student a hint 8 1.09 0.27 7 0.93 0.73 15 1.02 0.52 
or suggestion 
Gave student time to 8 1.06 0.18 7 0.79 0.81 15 0.93 0.56 
try again and self- 
correct 
Corrected student 7 0.31 0.37 7 0.50 0.65 14 0.40 0.51 
writing 
Graded student work 7 0.14 0.38 7 0.14 0.38 14 0.14 0.36 
Re-taught lesson 7 0.36 0.46 7 0.00 0.00 14 0.18 0.37 
segment 
Planned to review skill 7 0.50 0.65 7 0.21 0.39 14 0.36 0.53 
in future lessons 
Other 5 0.50 0.87 3 0.00 0.00 8 0.31 0.70 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Table D28 


Strategies Used when Discovered Student Misunderstandings about Writing (Pennsylvania - Social Studies) 


Module one Module two Overall 

Strategies n Mean = Std Dev n Mean = Std Dev n Mean = Std Dev 
Organized peer-editing 10 1.14 0.68 7 0.62 0.52 17 — 0.93 0.66 
session 
Scheduled in-class 9 0.87 0.79 7 0.81 0.74 16 0.85 0.75 
workshop time 
Held one-on-one 10 1.38 0.57 7 1.43 0.40 17 1.40 0.49 
conference with 
student 
Devoted time in lesson 10 0.78 0.72 7 0.67 0.62 17 0.73 0.66 
for students to use 
feedback 
Grouped students 10 0.63 0.76 7 0.24 0.42 17 0.47 0.66 
together on "need" 
basis for targeted 
instruction 
Modeled skill using 10 0.47 0.48 7 0.57 0.61 17 0.51 0.52 
my own writing 
Demonstrated skill 10 0.75 0.51 7 0.57 0.61 17 0.67 0.54 
using student's writing 
Provided grammar 10 ~=0.26 0.29 7 0.14 0.24 170.21 0.27 
mini-lessons 
Wrote specific 10 0.87 0.72 7 0.83 0.65 17 0.85 0.67 
comments on student 

work 

Had student revisit 10 0.88 0.79 7 0.79 0.58 17 0.84 0.69 
readings 
Offered student a hint 10 1.27 0.65 7 1.02 0.48 17 1.17 0.58 
or suggestion 
Gave student time to 10 1.23 0.61 7 0.93 0.64 17 1.10 0.62 
try again and self- 
correct 
Corrected student 10 0.93 0.68 7 0.40 0.41 17 0.72 0.63 
writing 
Graded student work 10 0.27 0.64 7 0.24 0.42 17 0.25 0.55 
Re-taught lesson 10 0.38 0.40 7 0.07 0.19 17 0.25 0.36 
segment 
Planned to review skill 10 0.85 0.82 7 0.24 0.30 17 0.60 0.71 
in future lessons 
Other 7 0.00 0.00 6 0.00 0.00 13 0.00 0.00 


Note. Scale is 0=not at all, 1=to some extent, 2=to a great extent. 
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Appendix E: Kentucky Matching and HLM Student Effect Estimates Tables 


Table E1 


Prior Achievement and Demographic Characteristics of Eligible and Matched LDC Treatment and 
Comparison Groups for KPREP Reading Outcome Analysis 


Eligible Sample Matched Sample 
a LDC Comparison LDC Comparison 
Student Characteristics (n=2,529) (n=43,333) (n=2,215) (n=13,934) 
Prior Achievement 0.167 -0.010 0.143 0.133 
Female (%) 49.1 49.2 49.8 49.9 
White (%) 90.5 82.0 91.9 91.8 
Hispanic (%) 2.6 3.5 2.4 2.2 
Black (%) 2.6 11.1 2.3 4.3 
Asian (%) 1.8 1.1 1.0 0.4 
English language learner (%) 0.4 1.4 0.4 0.3 
Special education (%) 10.4 11.4 8.6 8.2 


Free/reduced price lunch (%) 46.5 57.6 46.0 55.6 


Table E2 


Prior Achievement and Demographic Characteristics of Eligible and Matched LDC Treatment and 
Comparison Groups for KPREP Social Studies Outcome Analysis 


Eligible Sample Matched Sample 
a LDC Comparison LDC Comparison 

Student Characteristics (n=2,529) (n=43,333) (n=2,236) (n=17,726) 
Prior Achievement 0.176 -0.010 0.169 0.150 
Female (%) 49.1 49.2 48.7 48.7 
White (%) 90.5 82.0 92.2 92.2 
Hispanic (%) 2.6 3.5 2.1 2.1 
Black (%) 2.6 11.1 2.5 4.0 
Asian (%) 1.8 1.1 0.7 0.7 
English language learner (%) 0.4 1.4 0.3 0.3 
Special education (%) 10.4 11.4 9.8 8.7 


Free/reduced price lunch (%) 46.5 57.6 46.6 46.6 


Table E3 


Prior Achievement and Demographic Characteristics of Eligible and Matched LDC Treatment and 
Comparison Groups for KPREP Writing Outcome Analysis 


Eligible Sample Matched Sample 
_ LDC Comparison LDC Comparison 
Student Characteristics (n=2,529) (n=43,333) (n=2,252) (n=11,720) 
Prior Achievement 0.161 -0.009 0.138 0.145 
Female (%) 49.1 49.2 49.0 49.2 
White (%) 90.5 82.0 92.0 91.7 
Hispanic (%) 2.6 3.5 2.3 1.5 
Black (%) 2.6 11.1 2.5 4.6 
Asian (%) 1.8 1.1 0.9 0.9 
cas a 0.4 1.4 0.3 0.3 
Special education (%) 10.4 11.4 10.2 8.7 
Eres /ueduced pive 46.5 57.6 48.0 48.3 


lunch (%) 


Table E4 
2012-13 LDC Student Effect Estimates on K-PREP Reading 


Fixed Effect Model 1 Coefficient Model 2 Coefficient 


(S.E.) 


(S.E.) 


Level 1 Variables 

Female 

White 

Hispanic 

Black 

Asian 

English language learner 
Title I 

Special education 
Free/reduced price lunch 
Prior achievement 

Level 2 Variables 

LDC treatment 

Teacher effectiveness 
Missing teacher effectiveness 


School effectiveness 


0.146 (0.010)* 
0.045 (0.030) 
0.130 (0.043)* 
0.080 (0.037)* 
0.122 (0.094) 
0.001 (0.096) 
-0.061 (0.020)* 
-0.113 (0.025)* 
-0.107 (0.012)* 


0.687 (0.008)* 


0.081 (0.024)* 
0.279 (0.216) 
-0.029 (0.018) 


0.481 (0.142)* 


0.148 (0.016)* 
0.041 (0.042) 
0.127 (0.059)* 
0.081 (0.054) 
0.119 (0.143) 
-0.054 (0.140) 
-0.059 (0.028)* 
-0.109 (0.035)* 
-0.106 (0.017)* 


0.691 (0.012)* 


0.061 (0.018)* 
0.233 (0.214) 
-0.015 (0.012) 


0.241 (0.103)* 


Note. *significant at p =.05 


Table E5 


2012-13 LDC Student Effect Estimates on K-PREP Reading, 
Including Interactions with Prior Teacher Effectiveness and 
Student Characteristics 


Fixed Effect Model 1 Coefficient Model 2 Coefficient 


(S.E.) 


(S.E.) 


Level 1 Variables 

Female 

White 

Hispanic 

Black 

Asian 

English Language Learner 
Title I 

Special education 
Free/reduced price lunch 
Prior achievement 

Level 2 Variables 

LDC Treatment 

Teacher Effectiveness 


School Effectiveness 


LDC treatment by Teacher 
Effectiveness 
LDC treatment by Student 


Characteristics Interactions 


Female 
Special education 
Free/reduced price lunch 


Prior achievement 


0.147 (0.011)* 
0.046 (0.029) 

0.130 (0.042)* 
0.080 (0.037)* 
0.121 (0.093)* 
0.001 (0.095) 

-0.068 (0.020)* 
-0.095 (0.027)* 
-0.113 (0.013)* 


0.683 (0.009)* 


0.074 (0.031)* 
0.396 (0.231) 
0.467 (0.143)* 


-0.169 (0.513) 


-0.006 (0.018) 
-0.181 (0.045)* 
0.078 (0.022)* 


0.049 (0.014)* 


0.149 (0.017)* 
0.042 (0.042) 
0.128 (0.059)* 
0.082 (0.054) 
0.118 (0.143) 
-0.055 (0.140) 
-0.066 (0.027)* 
-0.087 (0.039)* 
-0.115 (0.018)* 


0.686 (0.013)* 


0.058 (0.023)* 
0.317 (0.227) 
0.239 (0.102)* 


-0.181 (0.202) 


-0.004 (0.017) 
-0.110 (0.034)* 
0.053 (0.017)* 


0.034 (0.011)* 


Note. *significant at p =.05 


Table E6 
2012-13 LDC Student Effect Estimates on Social Studies K-PREP Scores 


Fixed Effect Model 1 Coefficient Model 2 Coefficient 


(S.E.) 


(S.E.) 


Level 1 Variables 

Female 

White 

Hispanic 

Black 

Asian 

English language learner 
Title I 

Special education 
Free/reduced price lunch 
Prior achievement 

Level 2 Variables 

LDC treatment 

Teacher effectiveness 
Missing teacher effectiveness 


School effectiveness 


-0.109 (0.008)* 
-0.039 (0.030) 
-0.044(0.051) 

-0.035 (0.035) 
-0.010 (0.134) 
0.144 (0.079) 

-0.001 (0.027) 
-0.152 (0.025)* 
-0.099 (0.012)* 


0.680 (0.012)* 


-0.028 (0.025) 
0.066 (0.241) 
-0.034 (0.022) 


0.332 (0.113)* 


-0.108 (0.013)* 
-0.051 (0.044) 
-0.055(0.072) 
-0.048 (0.050) 
-0.081 (0.167) 
0.158 (0.109) 
0.039 (0.034) 
-0.144 (0.032)* 
-0.100 (0.017)* 


0.684 (0.022)* 


-0.002 (0.016) 
0.302 (0.085)* 
-0.028 (0.014) 


0.112 (0.081) 


Note. *significant at p =.05 


Table E7 


2012-13 LDC Student Effect Estimates on Social Studies K-PREP Scores, 


Including Interactions with Prior Teacher Effectiveness and 


Student Characteristics 


Fixed Effect 


Model 1 Coefficient 


(S.E.) 


Model 2 Coefficient 


(S.E.) 


Level 1 Variables 
Female 

White 

Hispanic 

Black 

Asian 

English language learner 
Title I 

Special education 
Free/reduced price lunch 
Prior achievement 

Level 2 Variables 

LDC treatment 

Teacher effectiveness 


School effectiveness 


LDC treatment by teacher 
Effectiveness 

LDC treatment by Student 
Characteristics Interactions 


Gender 
Special education 
Free/reduced price lunch 


Prior achievement 


-0.110 (0.009)* 
-0.040 (0.031) 
-0.044 (0.051) 
-0.035 (0.035) 
-0.012 (0.134) 
0.144 (0.080) 

-0.009 (0.026) 
-0.152 (0.027)* 
-0.105 (0.013)* 


0.675 (0.013)* 


-0.084 (0.031)* 
0.062 (0.243) 
0.339 (0.116)* 


-0.155 (0.254) 


0.016 (0.018) 
0.011 (0.046) 
0.073 (0.025)* 


0.075 (0.016)* 


-0.109 (0.014)* 
-0.052 (0.043) 
-0.056 (0.072) 
-0.049 (0.050) 
-0.080 (0.168) 
0.155 (0.110) 

0.023 (0.032) 

-0.141 (0.035)* 
-0.106 (0.019)* 


0.677 (0.023)* 


-0.026 (0.023) 
0.316 (0.084)* 
0.130 (0.081) 


-0.288 (0.082)* 


0.013 (0.016) 
-0.007 (0.037) 
0.039 (0.019)* 


0.050 (0.017)* 


Note. *significant at p =.05 


Table E8 
2012-13 LDC Student Effect Estimates on Writing K-PREP Scores 


Fixed Effect Model 1 Coefficient Model 2 Coefficient 


(S.E.) 


(S.E.) 


Level 1 Variables 

Female 

White 

Hispanic 

Black 

Asian 

English language learner 
Title I 

Special education 
Free/reduced price lunch 
Prior achievement 

Level 2 Variables 

LDC treatment 

Teacher effectiveness 
Missing teacher effectiveness 


School effectiveness 


0.428 (0.013)* 
0.050 (0.044) 
0.153 (0.060)* 
0.089 (0.049) 
0.245 (0.098)* 
-0.133 (0.130) 
0.066 (0.048) 
-0.424 (0.023)* 
-0.197 (0.013)* 


0.428 (0.007)* 


0.021 (0.045) 
0.459 (0.145)* 
-0.072 (0.029)* 


0.948 (0.247)* 


0.492 (0.024)* 
0.055 (0.071) 
0.117 (0.089) 
-0.028 (0.079) 
0.459 (0.221)* 
-0.171 (0.136) 
0.002 (0.072) 
-0.436 (0.036)* 
-0.149 (0.027)* 


0.428 (0.012)* 


0.014 (0.029) 
0.286 (0.111)* 
-0.051 (0.029) 


0.210 (0.143) 


Note. *significant at p =.05 


Table E9 


2012-13 LDC Student Effect Estimates on Writing K-PREP Scores, 
Including Interactions with Prior Teacher Effectiveness and 
Student Characteristics 


Fixed Effect Model 1 Coefficient Model 2 Coefficient 


(S.E.) 


(S.E.) 


Level 1 Variables 
Female 

White 

Hispanic 

Black 

Asian 

English language learner 
Title I 

Special education 
Free/reduced price lunch 
Prior achievement 

Level 2 Variables 

LDC treatment 

Teacher effectiveness 


School effectiveness 


LDC treatment by teacher 
effectiveness 
LDC Treatment by Student 


Characteristics Interactions 


Female 
Special education 
Free/reduced price lunch 


Prior achievement 


0.428 (0.014)* 
0.050 (0.044) 
0.153 (0.060)* 
0.089 (0.049) 
0.245 (0.098)* 
-0.133 (0.130) 
0.065 (0.049) 
-0.427 (0.025)* 
-0.199 (0.013)* 


0.427 (0.008)* 


0.023 (0.053) 
0.537 (0.156)* 
0.934 (0.253)* 


0.054 (0.291) 


-0.003 (0.028) 
0.036 (0.060) 
0.020 (0.032) 


0.011 (0.017) 


0.498 (0.027)* 
0.056 (0.071) 

0.118 (0.089) 

-0.026 (0.079) 
0.462 (0.218)* 
-0.173 (0.135) 
-0.019 (0.072) 
-0.443 (0.040)* 
-0.150 (0.031)* 


0.425 (0.013)* 


0.030 (0.042) 
0.362 (0.124)* 
0.242 (0.145) 


0.004 (0.120) 


-0.032 (0.031) 
0.031 (0.047) 
-0.002 (0.027) 


0.016 (0.016) 


Note.*significant at p =.05 


Appendix F: 
Regression Analyses of LDC Effects in Pennsylvania 


As noted in the body of the report, access to individual data on students’ performance on 
the Pennsylvania state assessment was restricted to pre-post data for the study year for LDC 
students in our sample and for students in comparable districts within the local region. Our 
analyses thus were severely constrained by the limitations of available data and are subject to 
numerous validity threats. The analyses used Coarsened Exact Matching (CEM) to compose the 
best available comparison group from the available data and applied available controls to the 
extent possible, but our results should be considered highly tentative and any inferences subject 
to further study. 


Teacher and student sample. As described in the body of the report, the LDC teacher 
sample for Pennsylvania included eight teachers new to LDC as well as 16 teachers with at least 
one prior year of LDC experience. Given the small number of teachers and students, we decided 
to include all 24 eighth-grade social studies and science teachers in the regression analysis to 
maximize sample size. In contrast to the Kentucky approach, available data did not enable the 


analyses to control for prior teacher effectiveness. 


The eligible LDC student sample for the analysis includes all students (a) who were 
enrolled in an eighth-grade social studies or science class taught by one of the 24 teachers, and 
(b) for whom prior and current achievement scores were available. This sample includes 1404 
students, as described in Chapter 2 of this report. As noted there, these students are roughly 
similar to all students statewide in the proportion of White students but show differences in 
representation of various minority groups. Notably the representation of Hispanic students is 
higher in the LDC sample and the proportion of economically disadvantaged students somewhat 
lower. Relative to student achievement variables, the LDC student sample performed slightly 
higher than students statewide in the prior year, as described in Chapter 2 (statewide means were 


not available for the outcome year). 


Standardized student scaled scores in Grade 8 reading for the 2012—2013 school year 
served as the outcome of interest. Prior scores in seventh grade in reading and math for the 
2011-2012 school year were used as matching variables and as covariates in regression analyses. 
Available student demographic variables served as additional matching and regression 


covariates. 


Available data included the six LDC districts with a total of 1446 with valid data at the two 
time points of interest, and two control districts with a total of only 738 students with valid data 
at the two time points. Coarsened Exact Matching, based only on individual-level variables, 


enabled a match of 1335 LDC students to 719 control students. The demographic and 


achievement data for these two groups, as seen in Table F1 below, show their close similarity. 


Table F1 


Pennsylvania Treatment and Control Group Characteristics, Based on 
Coarsened Exact Matching (n=1335 LDC Students, 719 Control Students) 


Characteristic LDC Control 
White (%) 65.0 65.0 
Hispanic (%) 23.4 23.1 
Black (%) 4.9 6.9 
Asian (%) 2.3 2.9 
English language learner (%) 3.0 3.1 
Free/reduced price lunch eligible (%) 31.6 31.6 
Special education (%) 9.4 9.4 
Mean seventh-grade math Z score .060 047 
Mean seventh-grade reading Z score 058 .030 
Mean eighth-grade reading Z score .054 011 


Regression analysis was used to investigate potential treatment effects on student 
outcomes. Due to available data, the analysis was limited to the individual level and could not 
take into account school or district effects. The same individual covariates and interaction 
variables from the Kentucky analyses were used for the Pennsylvania analysis. Results, shown in 
Table F2, reveal no treatment effect, nor treatment interaction effect. It is interesting, however, to 
see that the treatment/prior achievement interaction was significant at the .1 level and was 
consistent with the Kentucky finding. Regression results also indicate that prior achievement was 
positively related to reading performance and that English language learner status and special 


education status were associated with lower performance. 


Table F2 


Pennsylvania Regression Analysis of LDC Effect on State Reading Scores, CEM 
Matched Data 


Level 2 variable Model coefficient (SE) 
LDC treatment 0.026 (0.03) 
LDC treatment by student characteristics interactions 
Free/reduced price lunch eligible -0.011 (0.06) 
Prior achievement 0.049 (0.03) 
Student characteristics and prior achievement scores 
Grade 7 reading Z score 0.546 (0.03)*** 
Grade 7 math Z score 0.237 (0.02)*** 
White 0.048 (0.06) 
Hispanic 0.012 (0.07) 
Black 0.090 (0.08) 
Asian 0.164 (0.10) 
Special education -0.309 (0.04)*** 
Free/reduced price lunch eligible -0.144 (0.05)** 
English language learner -0.264 (0.07)*** 


**p = 01. ***p = 001. 


